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ABSTRACT 

In their search for new ways to assess their agencies' success 
in working with children and families, child welfare administrators and senior 
managers are increasingly seeking regular and reliable sources of information 
that help them evaluate agency performance, make ongoing decisions, and 
provide an accurate picture for agency staff and external stakeholders. This 
guide presents a framework for these quality assurance (QA) activities in 
child welfare agencies. The framework includes the broad elements all agencies 
should consider in creating new or energizing existing QA systems and is based 
on examples from ongoing QA efforts, existing federal requirements, research 
findings, and national QA standards from other settings. The framework is 
presented in implementation steps to assist administrators in conceptualizing 
the application within their agencies. Following the introduction describing 
the importance of QA for child welfare agencies. Step 1 details the activities 
involved in adopting outcomes and standards. Step 2 discusses ways to 
incorporate QA throughout the agency and includes strategic planning, creating 
a QA structure, and communicating quality expectations. Step 3 describes ways 
to gather data and information. Step 4 concerns procedures for analyzing data 
and information. Step 5 highlights how to use analyses and information to make 
agency improvements. The report's four attachments contain comprehensive case 
studies of state quality assurance systems, federal requirements for quality 
assurance systems, standards for quality assurance systems developed by 
national organizations, and resources for child welfare agencies. (KB) 
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Foreword 



Child welfare administrators and senior managers increasingly are searching for 
new ways to assess their agencies’ success in working with children and families. 
Rather than rely on anecdotal information or uneven data that may not apply to their 
entire systems, today’s child welfare managers seek regular and reliable sources of 
information that help them evaluate agency performance, make ongoing decisions, 
and provide an accurate picture for agency staff and external stakeholders. That is, 
today’s managers strive to take the initiative and push their own agendas rather than 
make reactive policy and practice changes in response to anecdotes, crisis cases, or 
uninformed external scrutiny and pressure. 

The term most often used to describe these efforts is Quality Assurance (QA). 
Many child welfare agencies have had entire QA divisions, or at least designated QA 
staff members, for a number of years. However, in practice, “quality assurance” has 
had no consistent meaning across child welfare agencies. Quality assurance efforts 
may range from administrative case review systems to periodic research studies to 
regular statistical compliance reports to comprehensive initiatives involving all these 
elements and more. 

The varied definitions also extend to the academic research and management texts 
focused on quality. An array of related and overlapping terms and techniques — 
including quality assurance, quality improvement, total quality management, quality 
assessments, performance measurement, evaluation research, goals, outcomes, 
standards, and indicators — sometimes just adds to the confusion. 

In this guide, the National Child Welfare Resource Center for Organizational 
Improvement (NCWRCOI) attempts to simplify the picture by presenting a frame- 
work for child welfare QA systems. While specific QA activities often will vary 
across agencies, the framework includes the broad elements all agencies should 
consider in creating new or energizing existing QA systems: The framework also 
presents the main elements in the form of implementation steps to assist readers in 
conceptualizing application within their agencies. 

The Resource Center created the framework based on examples from ongoing QA 
efforts in state child welfare agencies, existing requirements from Federal legislation 
and regulations, child welfare research and management studies, and national QA 
standards developed in other settings. Given these diverse sources of information 
and the increasing number of agencies adopting innovative QA strategies, the Re- 
source Center fully expects to enhance the QA framework in the coming years with 
even richer examples as the child welfare field learns more. This guide certainly is 
not the final word or necessarily the authoritative source on QA, but it attempts to 
pull together the important elements in one place for busy child welfare administra- 
tors and managers. 
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Isitr®claa©ti©ff& •— — ' Why Quality Assurance is Important 



Traditionally, quality assurance systems in child welfare agencies have focused on 
auditing case records to monitor and report on the extent of compliance with state 
and federal requirements. This focus paralleled federal approaches to monitoring 
state child welfare programs that often were directed towards assessing compliance 
with procedural requirements. Given the historic auditing focus in the child welfare 
field, small quality assurance staffs on the margins of agencies usually carried out 
the monitoring function, and their reports often had minimal impact on the services 
delivered by the agency. 

Today, many child welfare agencies are developing systems that move beyond 
compliance monitoring. These systems attempt to gather and assess a range of 
information on quality, and they work to implement needed improvements on an 
ongoing basis. As a way of differentiating these efforts from traditional compliance 
monitoring, the new approaches often are called continuous quality improvement 
systems. Moreover, recent federal initiatives, such as the new Child and Family 
Service Reviews, also support and encourage the move towards continuous quality 
improvement processes. 

The new approach improves upon traditional compliance monitoring in three 
ways. First, quality improvement programs are broader in scope, assessing practice 
and outcomes, as well as compliance. Second, they attempt to use data, information 
and results to affect positive changes in policy and case practice, along with compli- 
ance with federal, state and agency requirements. Third, these programs engage a 
broad range of internal and external partners in the quality improvement process, 
including top managers, staff at all levels, children and families served and other 
stakeholders. 

These characteristics of continuous quality improvement systems parallel the 
approach of the new federal Child and Family Services Reviews. Thus, states with 
strong quality assurance systems that focus on outcomes, involve a broad range of 
stakeholders, and facilitate continuous improvement will be better prepared for the 
new review process. In addition, they will be accomplishing the ultimate goal of the 
federal reviews, which is to continually improve outcomes for children and families. 

The National Child Welfare Resource Center for Organizational Improvement 
(NCWRCOI) has created the QA framework to reflect the current focus on develop- 
ing more comprehensive and effective quality improvement systems. This frame- 
work and implementation guide strips away the rhetoric and confusing terminology 
and presents straightforward information that will help agencies develop continuous 
quality improvement systems. 



The framework consists of five main steps. Each will be addressed in detail in this 
guide: 

H Step 1: Adopt outcomes and standards 
IS Step 2: Incorporate QA throughout the agency 
ID Step 3: Gather data and information 
U Step 4: Analyze data and information 

13 Step 5: Use analyses and information to make improvements 

Each section includes a description of the framework element, the specific tasks 
involved, and illustrative state examples of ways to accomplish the work. 

In addition to detailing the framework steps, the guide includes several attach- 
ments that provide excellent information and resources for states. The first attach- 
ment presents several comprehensive case studies of state quality assurance systems, 
as well as information on related components in other states. These case studies and 
examples illustrate a few of the varied QA approaches states have taken in recent 
years in an effort to improve their services and the outcomes for the children and 
families served. The second attachment summarizes the federal requirements for 
state quality assurance systems. The third attachment presents a variety of quality 
assurance standards that national child welfare organizations have developed in 
recent years. Finally, the fourth attachment presents an annotated bibliography of 
various quality assurance resources across a number of specific topics. 
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Step i — — Adopt Outcomes and Standards 



At the root of child welfare work are the goals agencies want to achieve with the 
children and families involved in their cases. The complexity of child welfare work, 
demands for public accountability, and new federal requirements increasingly have 
pushed agencies to make their goals an explicit part of their statewide strategic 
plans. From a quality assurance perspective, explicit goals are critical because they 
suggest the outcomes an agency intends to achieve with/for its clients. In turn, these 
client level outcomes suggest the key service level standards necessary to guarantee 
that children and families receive quality services to meet their needs. Therefore, 
these outcomes and standards provide the underpinning for the agency’s decisions 
about the types of quality assurance data and information to collect and analyze in 
Steps 3 and 4. This section provides examples of the types of outcomes and stan- 
dards child welfare agencies adopt and highlights their implications for quality 
assurance systems. 

Define Child Welfare Outcomes 

In recent years, a number of laws, regulations and initiatives on the federal and 
state levels increasingly require state child welfare agencies to define the outcomes 
they intend to achieve and then track performance on these outcomes over time 
through their QA activities. Most commonly, child welfare agencies have begun to 
focus on three broad outcomes: safety, permanency and well-being for children and 
families. Some examples of recent initiatives pushing these outcomes are the follow- 
ing: 

Child and Family Services Plan (CFSP): Federal regulations require that child 
welfare agencies, in order to be eligible for Title IV-B funds, develop and implement 
a five-year comprehensive child and family services plan. The plan must include 
goals expressed in terms of improved outcomes for the safety, permanency and well- 
being of children and families, and specific, measurable objectives that will be 
undertaken to achieve the goals. The plan also must describe methods used to 
measure annual progress towards meeting the goals and objectives, especially the 
outcomes for children, youth and families. 

Adoption and Safe Families Act (ASFA): In 1997, AS FA elevated the impor- 
tance of safety and expedited timeframes for the achievement of permanency for 
children served by the child welfare system. ASFA also required the U.S. Depart- 
ment of Health and Human Services (DHHS) to develop a set of outcome measures 
to assess state performance, and to report annually to Congress on state performance 
on these measures. The first annual report, released in August 2000, highlights data 
on state performance on seven outcomes related to safety and permanency, and 
thirteen associated performance indicators. Future reports also may include out- 
comes related to child and family well-being. 
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Child and Family Services Review (CFSR): In the summer of 1999, the 
Children’s Bureau began to implement a new federal review process for state child 
welfare agencies. Under the Child and Family Services Review process, state offi- 
cials, federal officials and community partners work together to assess state perfor- 
mance on seven outcomes and seven systemic factors supporting the achievement of 
the outcomes. The seven outcomes are divided into the broad areas of safety, perma- 
nency and well-being: 

Safety 

H Children are, first and foremost, protected from abuse and neglect 
H Children are safely maintained in their homes whenever possible and appropriate 

Permanency 

B Children have permanency and stability in their living situations 
B The continuity of family relationships and connections is preserved for children 

Child and Family Well-Being 

B Families have enhanced capacity to provide for their children’s needs 
11 Children receive appropriate services to meet their educational needs 
B Children receive adequate services to meet their physical and mental health needs. 



State Example — — Utah’s Performance Milestone Plan 

A strong feature of Utah’s quality improvement system is the clear outcomes and indica- 
tors that have been established in the state’s strategic plan. Based on these outcomes and 
indicators, Utah has created systems for regular tracking and reporting of related data and 
information. 

In May, 1999, the Utah Division of Child and Family Services (DCFS) released the Perfor- 
mance Milestone Plan, which describes in detail how the Division will improve services to its 
clients. The Performance Milestone Plan evolved out of court involvement in the child wel- 
fare system, but the Department views it as their business plan, which they intend to imple- 
ment with or without continuing court involvement or outside monitoring. Since 1994, the 
state has been working to comply with the monitoring requirements of the "David C. vs. 
Leavitt” Settlement Agreement. The Performance Milestones Plan was developed with the 
assistance of the Child Welfare Policy and Practice Group (CWPPG) after a 1998 court order 
directing the Department to design new, more valid and instructive measures of perfor- 
mance. 

The Performance Milestone Plan consists of 9 milestones which describe the activities 
the state will engage in to measure progress towards performance goals and make program 
adjustments based on feedback on how well the system is functioning. It defines the Division's 
outcomes and the indicators that will be tracked to measure progress on those outcomes. 
In addition, the Performance Milestone Plan describes the development and implementa- 
tion of a Practice Model to guide casework practice, defines the structures and processes 
that will be used to gather and analyze information from data and case reviews, and de- 
scribes the development of Quality Improvement Committees which will use information 
from these sources to guide necessary changes in the system. Many of the components of 
the plan have been implemented and are being used to make program improvements. 
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The seven systemic factors relate to the state agency’s capacity to deliver services 
leading to improved outcomes for children and families. Therefore, the CFSR 
process encourages states to examine these systems and make necessary improve- 
ments on a regular basis. The systemic factors are: 

ES Statewide information system 
0 Case review system 
a Quality assurance system 
m Staff training 
E Service array 

0 Agency responsiveness to the community, and 
M Foster and adoptive parent licensing, recruitment and retention. 

The review process defines the specific indicators that will be assessed for each 
outcome and for each systemic factor. These outcomes and indicators, along with 
those used by the Children’s Bureau in the Annual Outcomes Report, provide a good 
starting point for states in assessing their current focus and determining whether they 
need to modify the goals and outcomes underlying their child welfare systems. 



Define Practice Standards 

Adapting outcomes and indicators to your state’s system is one way of identifying 
key areas on which to focus through QA activities. In addition, many agencies take 
another step to ensure that these outcomes and indicators are communicated 
throughout their organizations: they develop practice standards that define the 
agencies’ expectations for day-to-day case practice. 



State Example — — Utah’s Practice Model - 

Utah also has a practice model that establishes a baseline for the Division of Child and 
Family Services’ interactions with families, community partners and staff. The model in- 
cludes seven practice principles and a set of key practice skills that are developed through 
training. For example, one of the principles is “partnership," and related skills are “engag- 
ing" children, families and other essential individuals. The Division is conducting training for 
all staff on the Practice Model, and as of May 2001 about 2/3 of the staff had received the 
training. The protocol for the qualitative case reviews Utah conducts reflects the practice 
principles. Therefore, the case reviews help communicate to staff the type of practice that is 
expected under the model and assess the extent to which it has been implemented. 



Defining and communicating practice standards can lead to an increase in compli- 
ance with requirements, improvements in case practice, and ultimately the achieve- 
ment of the agency’s outcomes. For example, a state might define a practice stan- 
dard as “engaging clients,” and then focus on ensuring that clients are invited to 
participate in case planning conferences, workers make required visits to homes, and 
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clients feel they are included in the process and their needs jure being addressed. The 
desired outcome of “engaging clients” in these ways might be reunifications happen- 
ing more quickly, and/or safer and more stable home environments that make 
reunifications less likely to be disrupted. 

Based on current knowledge in the field, as well as the various federal and state 
initiatives mentioned above, each state child welfare agency needs to decide what 
outcomes to emphasize and how to drive its practice accordingly to achieve them. 
One important method is covered in the next step of the framework: incorporating 
outcomes and the organizational QA structure into agency-wide plans. 



Incorporate QA Throughout the Agency 



The next step in building an ongoing quality improvement system is incorporating 
the main child welfare outcomes and indicators into the agency’s strategic plan, and 
creating a QA structure within the organization to facilitate the achievement of these 
outcomes and indicators. Creating a culture throughout the agency that supports 
quality improvement requires frequent, clear and consistent communication about 
agency expectations for performance on outcomes and compliance with practice 
expectations. The creation of this culture begins with top management s commit- 
ment to quality assurance. In addition, agencies should have dedicated quality 
assurance staff to work with internal staff and external stakeholders and to send a 
strong signal that quality improvement is an agency priority. 



Include QA Elements in Strategic Plan 

Federal requirements, national standards, and academic research all emphasize the 
importance of communicating quality improvement concepts through an agency’s 
strategic planning process. To comply with federal regulations around the Child and 
Family Services Plan (CFSP), states should engage in a strategic planning process to 
define the outcomes and practice standards they plan to track over time. The CFSP 
process calls for states to set long-term timetables and interim benchmarks for 
achieving their objectives. In support of these elements, the CFSP must include a 
description of the state’s quality assurance system and measures that will address 
any problems identified. Finally, federal regulations require states to submit annual 
progress and services reports (APSRs) in which they review and update their CFSPs 
where necessary. 

These federal regulations complement national standards for quality assurance 
systems that stress the importance of having a strategic planning process, and of 
developing both a long term plan and an annual operating plan (see Attachment on 
National Standards). The National Child Welfare Resource Center for Organiza- 
tional Improvement is in the process of developing a practice package on strategic 
planning that will provide a framework and numerous materials to help state child 
welfare agencies implement an ongoing planning process. Similarly, various aca- 
demic and management studies also discuss the need to establish child welfare 
program outcomes and continually evaluate program effectiveness (U.S. Department 
of Health and Human Services, 1997, Casey Outcomes and Decision Making 
Project, 1998, Young et al, 1994). 



Create a QA Structure 

In order to have an effective quality assurance process that monitors performance 
and supports quality, states need to involve a wide range of staff and organizations in 
quality assurance initiatives. Many states stress the responsibility of all managers 
and staff in the quality improvement process, and work to engage a broad range of 
personnel in the process. Many also work to engage external stakeholders, including 
other agencies and community members, in the work. The federal CFSP encourages 
such broad involvement in QA through its requirement that states conduct staff 
training to support the agency’s ability to meet its goals, and to involve major actors 
in the child and family service system in meeting the agency’s goals. 

Commitment to QA begins with top managers who are committed to the agency’s 
quality expectations. This commitment on the part of top managers, and their ongo- 
ing work to make quality a priority, is one of the major shifts necessary in moving 
an agency from a compliance monitoring focus to one of quality improvement. 
Lawrence Martin notes that all of the major American quality experts agree that the 
unqualified commitment of top management is absolutely essential to successfully 
promoting quality within human service agencies. He describes the need for manag- 
ers to provide leadership in creating a culture of quality by both setting the tone for 
change and taking specific actions designed to facilitate the transformation. Setting 
the tone can involve the use of themes, slogans and symbols which are repeated so 
often that they essentially become part of the organizational psyche. For example, 
the Maricopa County Department of Social Services in Arizona developed a vision, 
mission and values statement that can be found prominently displayed in offices, 
meeting rooms and hallways of the organization’s facilities. Specific actions can 
include requiring managers to report on quality improvements within the agency, 
and giving positive recognition to those that are implementing quality improvements 
(Martin, 1993). Simply put, if top managers are not committed to quality, the 
agency’s efforts to change the culture almost certainly will fail. 

Also, at a minimum, states should dedicate child welfare staff to QA initiatives in 
order to monitor performance. QA staff can work to ensure that people throughout 
the agency use information on quality, and to engage all staff in the process of 
examining data and acting to make improvements. Creating a separate, dedicated 
QA function in the organization sends a strong signal that quality is a major focus, 
and provides staff who can undertake and support the work throughout the agency 
on an ongoing basis. Generally, these staff will vary in their abilities and experience, 
depending on the QA activities the agency implements. Among the most critical 
characteristics necessary among various QA staff will be data analysis skills, an 
understanding of child welfare practice, work and processes, and an ability to trans- 
late and communicate the results of data analyses into formats that child welfare 
staff in the rest of the agency can understand and use to improve their work. 
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State Example — -Illinois’ Quality Improvement Structure 

The Illinois Department of Children and Family Services (DCFS) has staff and organiza- 
tions dedicated to quality assurance and quality improvement, and they draw on outcome 
data, case review results and consumer input to evaluate the quality of services. The Divi- 
sion of Quality Assurance, headed by an Associate Director for QA and Ql, oversees the 
quality improvement process. In this Office, there are 9 regional quality specialists and 10 
data analysts who are assigned to the state’s six regions to assist regional staff with the 
quality improvement process. There are also 9 staff in a field review unit, and four program 
analysts who compile data and produce reports at the state level. Staff who conduct special 
projects, and the Office’s managers make up the rest of this office. The regional quality 
specialists and the field review unit both participate in reviews and provide support to other 
staff involved in the quality improvement process. The program analysts who produce data 
on the state level and the data analysts assigned to regions both work to assure that data 
are available and utilized on the local level. 

Since purchase of service agencies provide over 75% of its services, the state has worked 
to improve its process of provider review and monitoring in recent years. On the state level, 
a Purchase of Service monitoring division was created that consists of the licensing unit, a 
field audit team, a provider technical assistance and training function, and agency perfor- 
mance teams (APTs). The APTs conduct monthly reviews of all agencies, reading cases and 
conducting satisfaction surveys. They also work to develop and analyze data profiles of 
contractors’ services and outcomes. The Purchase of Service Division joins the Division of 
Quality Assurance in providing oversight of the quality of services statewide. 

Illinois is taking the approach of involving all staff in the quality improvement process. 
Each team of employees takes on the role of a local quality improvement (Ql) team, and 
meets at least quarterly to review items on a standardized quality improvement agenda. 
These teams are part of a multi-tiered structure of Ql teams, where the local teams send 
representatives to the site Ql team, that team feeds into a regional quality council, and the 
regional quality councils are linked to the State Performance Support Team. This allows 
issues to be raised to the appropriate level. 

The items on the quality improvement agenda lead Ql teams and councils through a 
review of all of the information on quality — the results of peer review, consumer satisfaction 
data, data on incidents, accidents and grievances, program evaluation data and information 
generated by special projects such as accreditation. As the state continues to work to en- 
gage all staff in Ql, they are assisted by the commitment of the agency director to quality 
and accreditation. He has made it clear that the quality of agency services is a priority within 
the agency, and that he expects all offices and contractors to work towards a high standard 
of service. The Associate Director for Quality Assurance reports directly to the Director. 

The state also has engaged other organizational systems in focusing the agency on out- 
comes. Through the budget process, regional management agreements (RMAs) are devel- 
oped which include performance targets and actual regional performance data on specific 
indicators. The RMAs contain 25 specific measures for safety, permanency and well-being, 
and also have measures for other outcomes (including continuity of care) and system is- 
sues. For example, under child safety, one indicator in the agreement is the percent of child 
abuse and neglect investigations completed within 60 days. For one region, the target was 
98%, the YTD actual performance was 96.3%, and the historical baseline for the last two 
fiscal years was 86.7% and 90.8%. 
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Another factor to consider in creating a QA structure is the federal requirement 
that state quality assurance systems be in place across all jurisdictions providing the 
services detailed in the Child and Family Service Plan. In effect, states need to 
develop quality improvement processes that cover all regions of the state, and all 
groups of families served. Thus, states with quality assurance processes covering 
only limited groups of children or areas of the state need to expand their systems. 
Minimally, states should develop a plan to implement the quality assurance pro- 
cesses in a comprehensive manner over time, with a goal of eventually reaching all 
the jurisdictions in the state where services are delivered. For example, if a state uses 
a key quality review mechanism only for a targeted group of children (e.g., those in 
residential care), the state should expand the mechanism or develop new ones for 
other children served (e.g., those in foster care and/or at home). Similarly, if critical 
quality assurance data are being gathered or case reviews undertaken in a pilot area 
of the state or with a targeted group of providers, the state should expand the pilot to 
cover all areas and providers. 

Communicate Quality Expectations Throughout the Agency 

In addition to including QA elements and the QA organizational structure in the 
state mission and related plans, child welfare agencies should undertake several 
ongoing strategies for communicating quality expectations throughout the agency 
and the broader child welfare community: 

IS Incorporate expectations into training for new workers, existing staff and foster 
parents 

a Update policy and procedure manuals to reflect quality expectations 
H Include quality expectations in personnel performance evaluations 
0 Include quality expectations in budgets 

B Use existing case review processes to review for quality issues 
0 Include quality standards in licensing procedures 
B Include quality expectations and standards in provider contracts 

The next step in the framework describes the way QA staff gather data and infor- 
mation related to the state’s outcomes and indicators. 
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Gather Data and Information 



Once a child welfare agency has defined the key outcomes and indicators and 
created a QA culture, including a separate QA function within the organization, the 
QA system must gather a variety of relevant data and information. As in the previous 
steps of the framework, a number of federal regulations, state initiatives, and aca- 
demic studies help inform the quality improvement process. First, all these sources 
agree that child welfare agencies must use quantitative data to track both the status 
of clients and services and the program outcomes achieved. Second, QA systems 
should include a case review process. Third, they should implement strategies to 
listen to and involve the agency’s clients in assessing quality, as well as obtain input 
from external stakeholders in the community. Fourth, these systems will use a 
variety of other information at their disposal as a result of other review processes. 



Collect Quantitative Data 

Data used to monitor quality can be gathered through automated information 
systems, or through other reporting mechanisms in place within the agency. Two 
federal initiatives, the Child and Family Services Plan (CFSP) and the Child and 
Family Services Review (CFSR) process, make it clear that state quality assurance 
systems should be continually tracking data on both outcomes and systemic factors. 

Each state’s CFSP must describe how the state will produce valid and reliable data 
to determine whether the timetable for accomplishing its goals and objectives is 
being met. In addition, updated information must be obtained throughout the five- 
year plan period to measure progress in accomplishing the goals and objectives cited 
in the CFSP. 

Similarly, the CFSR process requires states to analyze data on their performance, 
and these data are used in determining the extent of the state’s substantial confor- 
mity to federal expectations. Rather than waiting for their federal review to occur, 
states can benefit from incorporating these same data into their ongoing QA activi- 
ties. For example, state QA systems should regularly examine statewide aggregate 
data elements obtained from the Adoption and Foster Care Analysis and Reporting 
Systems (AFCARS) and the National Child Abuse and Neglect Data System 
(NCANDS), especially those included in the data profiles provided to the state at the 
beginning of the federal review process. These data elements include: 

E3 Children entering care based on child abuse/neglect reports 
M Child fatalities 
E3 Recurrence of maltreatment 

El Incidence of child abuse and/or neglect in foster care 
0 Permanency goals for children in care 
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0 Number of placement settings 
M Number of removal episodes 

U Number of children in care 15 of most recent 22 months 

01 Median length of stay for foster care 

0 Length of time to achieve permanency goal 

Under two of the seven outcomes to be assessed in the CFSR process (safety 
outcome #1 and permanency outcome #1), there are six statewide data indicators on 
which state performance will be compared to a national standard in determining 
substantial conformity. States should pay particular attention to monitoring these 
statewide data indicators on an ongoing basis. These six statewide data indicators 
are: 

H Recurrence of maltreatment 

E3 Incidence of child abuse and/or neglect in foster care 
0 Foster care re-entries 
H Length of time to achieve reunification 
S3 Length of time to achieve adoption 
0 Stability of foster care placements 



State Example —Utah’s Use of Outcome Data 

One of the milestones in Utah’s Performance Milestones plan describes the 16 trend 
indicators that will be used to track the outcomes of protection, permanence and well-being 
for those served by the DCFS. Data on the trend indicators are drawn from the state’s 
SACWIS system, and are published in an Annual Outcomes Report. In this report, data on 
the indicators are used to measure progress towards specific performance goals under nine 
outcomes in the general categories of protection, permanency and well-being. The report 
provides multi-year trend data on each of the indicators, which allows the Division to track 
performance over time. The Annual Outcomes report also includes an appendix which re- 
ports on the state’s performance on the 13 indicators used in the national Annual Report to 
Congress, and recent trend data on those indicators. Recently, the state has also begun to 
produce parallel annual reports on the regional level, with data on the indicators and out- 
comes. 



The CFSR process also requires states to obtain supplemental data, other than the 
NCANDS and AFCARS profiles, to review the well-being outcomes and the sys- 
temic factors. Since data to assess well-being and the functioning of systems will be 
required during the regular review process, states also could benefit from establish- 
ing quality assurance processes to regularly obtain and evaluate information on these 



areas. 



In addition to these data elements included in the federal review process, state QA 
systems should collect and analyze data on a number of other areas. Although 
quality improvement systems move beyond mere compliance monitoring, state QA 
systems still must undertake. such monitoring in practice. Thus, many QA systems 
collect and monitor data reflecting the timeliness of work in various areas, including 
investigations, service plan development, home visits with parents and children, 
permanency hearings, and other information that helps managers and supervisors 
identify trends and adjust their staff’s work. 



State Example — Alabama’s Use of Data 

Alabama has geared its QA system to examine, report on and improve department perfor- 
mance under three outcomes — safety, permanency and well-being — and seven systemic 
factors — community collaboration, service array and resource development, individualized 
service plans, quality assurance and supervision, staffing and caseloads, staff and provider 
training, and information system capacity. The Department has developed 51 indicators for 
these outcomes and systemic factors, which are regularly assessed by the QA system. 

Alabama uses several types of information to assess these indicators, including: 

ES Quantitative and factual information to describe activities, service capacity and other 
measurable factors 

m Qualitative and outcome information 

@ Information obtained from community stakeholder interviews 

H Information related to compliance with federal, state and department program 
requirements. 

As these diverse sources of information indicate, the QA system relies on data gathered 
centrally from the state’s automated systems and locally from records kept on the county 
level to assess the outcomes and systemic factors. As part of the QA process, the state QA 
staff supplies counties with data on safety and permanency from the state’s information 
systems. Counties then supplement the data on some aspects of permanency and well- 
being, and also provide extensive data on systemic factors. For example: 

El On permanency, the state supplies data on length of stay and permanency goals for 
children in out of home care, while the county supplies the number of children 
placed in and out of county and the number committed to care by different agen- 
cies. 

0 Counties complete an education data chart on the educational status of children in 
the system. 

B The counties provide data on staff and provider training, including the number of 
staff and providers who have completed specific trainings and the number who 
need training. 
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Conduct Case Reviews 



Child welfare agencies conduct two primary types of QA case reviews: case 
record reviews and qualitative case reviews. Many states have found that involving 
child welfare staff, including caseworkers, supervisors and managers, makes their 
case review process more effective. In addition, child welfare agencies often benefit 
from involving external stakeholders in conducting qualitative case reviews. 

IS Case record reviews: Many states review case records to monitor the extent of 
compliance with requirements. The review process usually entails reading case 
records and applying a standardized case review instrument that specifies the 
items to be reviewed. In one approach, state review staff dedicated to this function 
conduct these reviews. Another approach that many states have found beneficial is 
to conduct peer reviews, where child welfare staff are involved in reviewing the 
work of their colleagues. 

0 Qualitative case review: In qualitative case reviews, review teams read the case 
record but also interview all those involved in the case to assess the quality of 
services provided — the child and family, foster parents or other care providers and 
others who are involved. These intensive, in-depth reviews usually are conducted 
on a small sample of cases within a local office or county. Review teams made up 
of state staff and a range of external individuals often conduct these reviews. 
Consultants or university staff sometimes provide professional reviewers. States 
also may include external stakeholders such as representatives from other service 
systems, the legislature and advocacy groups on the review teams. Finally, in 
some cases, community members serving on quality improvement committees 
conduct these qualitative case reviews. 

The new federal CFSR process also relies on in-depth, qualitative case reviews to 
assess performance on some outcomes. For example, the CFSR assesses Safety 
Outcome # 1 (“children are, first and foremost, protected from abuse and neglect”) 
through two statewide aggregate data indicators (the recurrence of maltreatment 
statewide and the incidence of child abuse/neglect in foster care statewide) and 
through two qualitative indicators reviewers evaluate on site for each case included 
in the sample. 

In developing case review systems to assess the quality of services, states also 
should consider systems that may already be in place. Under CAPTA requirements, 
states receiving grants for child abuse and neglect prevention and treatment pro- 
grams are required to develop citizen review panels to examine the state’s policies 
and procedures by reviewing specific cases if appropriate. States receiving commu- 
nity based family resource and support grants also have a requirement for a peer 
review process. Finally, states should look at their ongoing case review processes 
already used to conduct the required periodic reviews for children in out of home 
care. All of these systems may offer important lessons and or adaptable structures 
for states that decide to examine the quality of services through case reviews. 
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State Example —-—Case Reviews in Illinois 

Afield review unit in Illinois conducts case reviews, and an extensive peer review process 
examines compliance with key department policies and case practices. Field review staff 
from the Division of Quality Assurance conduct reviews of DCFS operations through on-site 
audits. Reviewers read and assess a random sample of cases for compliance with depart- 
ment policies and procedures, accreditation standards, as well as good casework practice. 
They share their findings with operations staff and discuss them in detail during regional 
reviews. 

Through the peer review process, caseworkers and supervisors conduct reviews of cases 
in which they are not involved. In each of the state’s 74 field offices, peer reviews occur 
every quarter of each year. The state aims to review 10% of the total cases served each 
year, and occasionally adds extra cases to the sample to ensure that there are at least two 
cases from each team reviewed each time. The peer review instrument focuses on assess- 
ing whether key policies and practices were followed. The results of the peer review are 
tabulated and shared at the field office level and at the regional level. In addition, the 
caseworkers and supervisors for the cases receive the results of the review. If there are 
areas where a case did not score well, an improvement plan must be developed for that 
case. 

The state also has begun conducting preparatory child and family services reviews that 
mirror the federal Child and Family Service Review process. These are qualitative reviews 
that involve reading the case record and interviewing all parties involved in the case. DCFS 
staff, along with staff from other agencies (such as university educational partners) are 
reviewing 120-150 cases in each region. The Department has contracted with a group at a 
university to assist with the reviews of contracted private providers. The reviews focus on 
assessing outcomes and the service delivery process, and result in written reports. The 
Division of Quality Assurance then works with the regions to develop action plans to make 
needed changes. 



Gather Input from Children and Families 

Qualitative case reviews usually involve interviews with the children and families 
being served, and their input helps determine the effectiveness of child welfare 
services. This emphasis on listening to children and families as part of the review 
process reflects a growing tendency to involve families in the process of planning 
and delivering services. Reforms like family-centered practice, family group confer- 
ences, strengths-based assessments and wraparound services reflect a shift in focus. 
Rather than merely seeing families served as clients to whom things are provided, 
child welfare agencies have begun to consider them as active consumers whose 
strengths and needs should help drive the agency. 

Thus, in addition to qualitative case reviews, many states use a variety of mecha- 
nisms to obtain input from the children and families served by the child welfare 
system. These include: 

H Discharge interviews with children and families 
SI Grievance/complaint mechanisms 

E3 Staff dedicated to assuring agency responsiveness to consumers 
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El Periodic focus groups 
0 Surveys. 

This emphasis on involving and listening to children and families also corre- 
sponds to the way many other public organizations and businesses are assessing 
quality. For example, one of the primary shifts in recent years in the business world 
has been towards total quality management (TQM). TQM is defined as a new way of 
thinking about the management of agencies, and one of its primary characteristics is 
a total commitment to customers. Organizations gather regular feedback and data on 
customer satisfaction, and consumers are active participants in the agency’s quality 
programs (Martin, 1993 and Gunther and Hawkins, 1996). 



State Example — —Missouri’s Children and Family Surveys 

In Missouri, the Division of Family Services realized that to build a total quality organiza- 
tional culture, it needed feedback from the children and families whom it served. A team of 
participants representing all sections and levels of the organization worked together to 
develop a survey mechanism. As a result, five survey instruments were designed to target: 
1) adults recently receiving Child Protective Services, 2) adults served through Family-Cen- 
tered Services, 3) adults served through Intensive Family Preservation Services, 4) Foster/ 
Kinship Care providers, and 5) youth in Out-of-Home Care. Each survey addresses broad 
consumer issues such as participation in the service delivery process, how they feel they 
were treated, if they feel their needs were met, and the availability of agency staff. In addi- 
tion, each survey contains items that address the specific needs of each targeted respon- 
dent. 

An automated system generates the surveys and they are sent via mail at specific points 
in time during and after service delivery. Each survey includes a self-addressed, stamped 
envelope to facilitate a higher response rate and ensure confidentiality. Information from all 
returned surveys is entered into a database, aggregated, and provided in report form for 
review in the Continuous Quality Improvement Meeting process. The agency expects the 
survey information to assist staff in responding to consumer needs and in providing the 
best quality services possible. 



Gather Input from External Stakeholders 

Many states have built the involvement of external stakeholders into their ongoing 
quality assurance processes. In addition to interviewing some external stakeholders 
during qualitative case reviews, many states involve stakeholders through other QA 
mechanisms: 

0 Stakeholders serve on quality improvement committees 
H QA staff conduct stakeholder interviews, focus groups, and/or surveys 
ES External review organizations participate in agency activities 
0 Dedicated staff respond to external reviews. 
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These activities also help states meet the CFSP and CFSR requirements for 
external stakeholder consultation. For example, each state’s CFSP must describe the 
ongoing consultation process that will ensure the continued involvement of a wide 
range of major actors in meeting the agency’s goals and objectives over the five-year 
period of the plan. In addition, the CFSR encourages states to continue consultations 
with these stakeholders on an ongoing basis. The CFSR measures the extent of 
consultation directly under the systemic factor of “agency responsiveness to the 
community.” 

The literature also offers persuasive arguments about the benefits of involving 
external stakeholders in the QA process. The child welfare field increasingly recog- 
nizes that child welfare agencies cannot, on their own, protect children at risk of 
abuse or neglect. By joining in collaborations with other service providers and 
community based organizations, child welfare agencies can leverage more of the 
necessary resources to address the needs of children and families. Similarly, since 
lack of resources often stymies the successful implementation of service plans, 
collaborative efforts to enhance service networks are critical to effective services 
(Farrow, 1997). 



State Example — “—Stakeholder Involvement in Utah 

Under its Performance Milestone Plan, Utah has established Regional and State Quality 
Improvement Committees. Along with management at the state and county level, these 
committees work to review information on quality and identify needed improvements. 

The Regional Committees have up to ten members, including two Division staff— the 
Associate Regional Director and the regional data person— and up to eight community mem- 
bers. Community members have a leading role on these committees, charged with studying 
the data and outcomes children, families and communities experience, and suggesting 
changes in resource deployment, policy, procedures and practice that will improve or main- 
tain favorable outcomes. Some of the Regional Quality Improvement teams have been very 
active in reviewing information on quality and working for needed improvements. 

Utah obtains input from the children and families served through the qualitative services 
reviews. These are intensive reviews of a small sample of cases that involve a review of 
case records and interviews with all parties involved in each case. The state involves other 
stakeholders in conducting these reviews and as members of the quality improvement com- 
mittees. The state has also created a staff position titled "Constituent Services Specialist" 
who is charged with streamlining the process for complaints from consumers and working to 
integrate the recommendations of external review bodies into the ongoing work of the Divi- 
sion. This person has worked extensively with regional administrators to assure that issues 
or problems experienced by those served are addressed, and that a response is made at 
the lowest possible level. 
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Use Other Available Information 



Finally, the quality assurance process should draw on information generated 
through a variety of other sources that might reflect on the quality of services and 
outcomes. For example: 

E Internal and external evaluations of agency programs 
B Evaluations of staff/provider training sessions 
B Legislative audits 
S Reports from citizen review boards 
11 Child fatality review team results 

The next step in the framework focuses on analyzing the types of data and infor- 
mation covered in this section. 
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Analyze Data and Information 



As the previous section illustrates, states rely on a vast array of data and informa- 
tion to assess the quality of their services and the outcomes achieved for children 
and families. Given these diverse data and sometimes complex sources, state QA 
staff often need to take the lead in developing analyses and reports that help translate 
the results into understandable and relevant information. However, as with the other 
steps in the QA framework, state child welfare agencies should include a variety of 
staff and other stakeholders in the data analysis process itself. This ensures an 
inclusive process and improves the data interpretation and adjustment of agency 
activities in response to the findings. 



Involve Varied Staff in Analyzing Information 

States with a quality assurance structure usually have both dedicated quality 
assurance staff and a wide range of other staff and organizations involved in analyz- 
ing information on the quality of child welfare services. Models for dedicated 
quality assurance staff include the following: 

m Office of Quality Assurance, with Regional Quality Specialists: Many states 
have state level staff dedicated to quality improvement activities. In addition, 
these states often have staff assigned to regions, counties or local offices. These 
local staff — called, for example, quality specialists or quality assurance coordina- 
tors — often play a key role in working to encourage staff to examine and act on 
data; organizing staff and others to conduct case reviews; overseeing stakeholder 
input; and staffing local quality teams, councils or committees. Often the state 
level office provides support to the regional staff, and takes an active role in 
organizing the quality improvement process. In some states, the state office 
organizes reviews, or works closely with the unit that manages the case review 
process. These offices often are headed by Deputy or Associate Directors of the 
agency who report directly to and work extensively with top agency management. 

EH Data Analysts: Staff who work with information and reporting systems and 
produce and distribute data reports are a key part of quality improvement systems. 
These staff sometimes are part of a state QA division, or they may be organiza- 
tionally separate but work closely with QA staff and organizations. These data 
analysts, or their managers who direct and review their work, must have a thor- 
ough understanding of the agency’s case practice in order to translate QA data 
into clear and relevant reports and analyses. 

H State, Regional and Local Quality Improvement Teams/Councils: These exist 
in states that require all staff members to be on a site or unit quality improvement 
team. Usually, the site team sends members to a local or regional quality improve- 
ment team or council, which in turn sends members to a statewide quality im- 
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provement council. Issues identified at any level can be addressed at that level, or 
raised to a higher-level team for their consideration. The teams at each level must 
meet regularly and cover points on a standard quality improvement agenda. The 
teams review available quality data, and plan and work towards implementing 
needed improvements. The regional and state level teams often include represen- 
tatives from the community in addition to child welfare staff. 

0 State and Local Quality Improvement Committees: Quality improvement 
committees often are composed primarily of stakeholders from other service 
providers and from the community. Sometimes, data analysts or quality assurance 
coordinators will be members of these groups, and other times they will assist 
these groups as staff who help organize the meetings, provide materials, and work 
to ensure follow up. These committees are often charged with examining informa- 
tion on quality and working for improvements in the child welfare system. Some- 
times the committee members are involved in gathering quality data by conduct- 
ing qualitative case reviews or stakeholder interviews. 

The quality assurance structure often involves other staff and organizations in 

analyzing and using information on quality, including: 

0 Administrators and managers: The Director of the Department can play a major 
role in the quality improvement process by establishing quality as a goal and 
reinforcing it through his or her expectations of managers. Managers at the state, 
regional, local and unit level — from Regional Administrators to unit supervi- 
sors — play a critical role by examining information on quality and working in 
conjunction with the QA staff, committees or teams to identify and address areas 
for improvements. 

@ Case Review Units: Some child welfare agencies locate staff involved in con- 
ducting case reviews in distinct units apart from the quality assurance staff. 
Nevertheless, the quality assurance staff and organizations can use these case 
reviews as a critical source of information. These case review units may focus on 
department level case reviews and/or the required periodic reviews for children in 
out-of-home care. 

H External stakeholders and community members: Representatives from exter- 
nal stakeholder groups, such as other service systems, the courts, the legislature or 
advocacy groups, may participate in review teams for qualitative case reviews, or 
serve on quality improvement committees 

0 All staff: Some agencies see the continuous quality improvement process as an 
agency-wide effort. Thus, every employee serves on a quality team which meets 
regularly to review quality data and plan and implement needed improvements. In 
this approach, quality is not something to be pursued merely by the QA staff and 
top management, but instead is an expectation of every staff member. 

B Consultants: Some states use consultants to help organize a qualitative review 
process or other evaluations. These consultants provide professional reviewers 
who work with state staff and stakeholders and train them to conduct qualitative 
case reviews and utilize data on an ongoing basis. 
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H University staff: University researchers or other staff may help organize a case 
review process, conduct surveys or focus groups of children, families or other 
stakeholders, and/or develop data analysis techniques in concert with state QA 
staff. 



State Example Alabama’s QA Staff/Structure 

Many staff and organizational units in Alabama participate in gathering and analyzing 
information on quality. The system in Alabama has three components: (1) county quality 
assurance systems, (2) the Office of Quality Assurance in the State Department of Human 
Resources, and (3) the State QA Committee. 

County QA systems consist of a QA coordinator and a county QA Committee. There are 67 
counties in Alabama, and each has an allocated position for a QA coordinator. Counties with 
populations larger than 80,000 have a full time QA coordinator, and smaller counties have a 
half-time position. Coordinators’ responsibilities include organizing and supporting the county 
QA committees, collecting and evaluating information, and issuing the required quarterly 
and annual quality assurance reports. 

Each county has a county QA Committee appointed by the County Director of Human 
Resources. These county QA committees primarily include representatives of past service 
consumers, service providers, other public and private agencies, allied professionals, the 
courts and community stakeholders. The County QA coordinator serves on the Committee 
as a liaison to the agency, and occasionally other staff may also join to help integrate QA 
functions into the Department’s on-going service delivery process. 

In Alabama, these community-based QA Committees are responsible for conducting in- 
tensive, qualitative case reviews on a minimum number of cases annually and making rec- 
ommendations back to the Department. The county QA committees also are responsible for 
holding regular meetings, routinely reviewing data related to the outcomes and systems, 
and conducting special studies of issues raised by the case reviews and data. The Commit- 
tee also participates by providing input to and approving the county QA reports made to the 
state, and advocates on behalf of the agency, consumers or providers for issues related to 
improving services, agency capacity or outcomes. 

The Office of Quality Assurance (OQA) includes a Program Manager, Program Supervisor 
and five Quality Assurance Specialists. This state level office supports the State QA Com- 
mittee, and assists counties in developing and maintaining quality assurance functions. 
OQA works closely with the one staff person in the Office of Data Analysis who produces 
data from state systems for the counties. The OQA analyzes data, and conducts on-site 
reviews. 

Stakeholders are involved in assessing the quality of child welfare services both through 
the county quality assurance committees and through the State QA committee. The State 
QA committee includes representatives from twenty specific organizations and entities that 
make up the child and family service delivery system. The Committee has responsibility for 
monitoring outcomes and agency performance from a statewide perspective and for facili- 
tating the development of and networking between county QA committees. The State QA 
committee also serves as a link between the community and the State Department of 
Human Resources, and works to promote an effective child welfare system that supports 
positive outcomes. Specifically, the Committee receives information from county QA reports, 
state QA review reports and statewide QA reports, initiates special studies to investigate 
issues raised by the reports or information received from other sources, and makes recom- 
mendations for improvements. 
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Translate Data and Information into Quality Assurance Reports 

Quality assurance staff and others involved in promoting quality services need to 
analyze information gathered on quality and produce understandable reports. These 
QA reports should illustrate the agency’s performance and help other staff in the 
agency plan and make necessary improvements. The basis for these reports will be 
the types of data and information detailed in Step 3: data gathered through auto- 
mated information or other reporting mechanisms; results of case reviews; input 
from children, families and external stakeholders; and information from a variety of 
other sources that QA staff may include in the analysis. 

The capability of state automated information systems to track and report relevant 
information on quality is expanding. While some automated systems produce reports 
mainly at the state level, a growing number produce regional, local, unit and even 
worker level reports. In addition, some states have systems that allow staff at the 
state, regional or local levels to customize their own reports, and have taken other 
steps to encourage staff at all levels to use data as part of their ongoing work. The 
staff involved in designing and promoting the use of reports from automated sys- 
tems — sometimes called data analysts — are key partners in quality assurance sys- 
tems. Quality assurance staff and data analysts must work together to ensure that 
staff within the agency can understand QA reports and their implications for the 
quality of services and/or the outcomes among children and families. 

The main types of QA reports from automated systems that states have found to 
be useful are the following: 

0 Outcome reports: These focus on agency outcomes. Examples include lengths of 
stay for children in out of home care or foster care reentry rates. 

E Practice reports: These focus on key practice issues that can be gleaned from 
automated or other reporting mechanisms. An example is the number or percent- 
age of cases where family team meetings were held. 

Si Compliance reports: These reports provide information on the extent of compli- 
ance with agency requirements. For example, reports might indicate the percent- 
age or number of cases with permanency goals established within required time 
frames, or the percent of investigations completed within required time frames. 

Some examples of formats that have been helpful include: 

SI Reports that allow easy comparison across regions, local offices, and units. 

0 Reports on exceptions, such as reports flagging cases where investigations are 
past due. 

1! Early warning reports identifying cases that do not meet requirements prior to a 
review. For example, a report may list all cases that do not have permanency goals 
developed within required time frames. 
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State Example — — Alabama’s QA Reports 

Each county QA coordinator in Alabama, with input from the County QA Committee, puts 
together written Quality Assurance reports on a quarterly basis, and produces an annual 
report at the end of each year. Among the people and units who review, or have the opportu- 
nity to review, these reports are the state Office of Quality Assurance, other state office 
staff, the Federal Court Monitor, the Plaintiffs attorney and the State QA Committee. 

The Quality Assurance reports provide a structure for pulling together state and county 
data, and they ensure the regular evaluation of data under each of the outcomes and sys- 
temic factors. Each of the areas in the reports includes a “Discussion/Analysis of the Data" 
narrative section, through which the county must answer specific questions that involve 
analyzing the state and county data in the report. For example: 

Q For children with a length of stay of 13 consecutive months, summarize the barriers 
to achieving permanency and describe efforts underway to ensure that children 
achieve their permanency goals in a timely manner. 

0 Describe the reasons children identified as needing special education services did 
not receive such services during the reporting period. 

0 Describe any training needs that were identified during the reporting period and how 
these needs are being addressed. 

The state Office of Quality Assurance conducts on-site reviews of the County Depart- 
ments of Human Resources. During these reviews, state staff uses the qualitative case 
review protocol to review a small sample of cases and also validates the county QA 
committee’s previous reviews by re-reviewing a small sample of the same cases. They also 
interview community stakeholders to assist in evaluating the status of outcome and sys- 
temic issues, with a particular focus on evidence of a functioning County QA Committee. The 
OQA reviewers use the findings from this review, in conjunction with information from the 
county’s QA report and the outcomes of county QA committee work (e.g. case reviews, 
special studies, etc.), to complete the “Summary of Findings and Recommendations Form,” 
which reports on the 51 key indicators of outcomes and systemic factors, highlighting 
strengths, areas needing improvement and necessary recommendations. 



The development of QA reports often will be an iterative process and usually will 
include a variety of staff members. Quality assurance staff and data analysts should 
work together to produce draft reports and validate the applicability of the data 
elements involved. They should test each report’s clarity and usefulness through 
consultation with agency administrators, managers and other staff and make changes 
based on their reactions and input. Once the data elements and format of the reports 
have been finalized, the QA staff can take the lead in producing the reports on a 
regular basis and ensuring their distribution to the relevant staff within the agency. 

In addition to reports developed using data from automated systems, QA staff also 
will utilize data and information from other sources that reflect on the agency’s 
quality of services. For example, if a case review system routinely reports on com- 
pliance with key policies or practice issues, these analyses should be used as part of 
the ongoing quality assurance system. 

On a system wide level, staff often examine data from a variety of sources to 
assess agency progress towards the outcomes and practice standards adopted and 
incorporated into the state plan. Quality assurance staff, or quality improvement 
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teams, usually have a regular process for analyzing information on quality from data 
systems, case reviews, stakeholder input and other sources. These include: 

E3 Quality improvement team meetings that follow a predefined agenda and involve 
reviewing quality information from a number of sources. 

H Quality assurance reports that pull together information from data, case reviews 
and stakeholder input to analyze the system’s strengths and weaknesses. 



State Example — — Analyzing Data in Utah 

DCFS staff in Utah review and use outcome data in two organized ways. The Division has 
established trend analysis teams on the state level who are charged with reviewing and 
assessing the meaning behind the data. The state also has information analysts on the 
state level who produce the reports, and work with data contacts in each region of the state. 
These regional data contacts have been producing reports on the outcomes and indicators 
in the statewide report on the local level, and working to facilitate the use of data. They 
serve on the Regional Quality Improvement Committees to assist them in their work of 
reviewing and analyzing trend data. 



Broadly, quality assurance efforts throughout the agency should ensure that all 
employees receive regular information about the quality of services. Regular and 
open communication to all levels about performance helps engage staff in efforts to 
improve the quality of services. The next section details ways to use the data analy- 
ses and information that result from the QA process to make improvements through- 
out the child welfare agency. 
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Use Analyses and Information 
to Make Improvements 



Quality assurance information from data, case reviews, stakeholder input and 
other sources must be fed back into the organization to make staff and managers 
aware of their performance. To make the enormous effort involved in creating and 
maintaining the QA system worthwhile, however, the agency must take a final step 
beyond merely distributing reports: it must use the information to plan and imple- 
ment improvements that will enhance the quality of services and ultimately the 
outcomes for children and families. 



Create Feedback Loops 

QA systems feed the results of QA processes and data analyses back to staff 

throughout the agency in a variety of ways. Some examples of information feedback 

loops that should help promote improvements in case practice, agency services, and 

systemic factors include: 

B Staff at all levels, including caseworkers, supervisors, regional or area managers 
and state level managers, receive regular data reports and use the information to 
plan and implement changes in practice. 

H During the case review process, reviewers meet with caseworkers and supervisors 
whose cases are being reviewed to debrief their findings and discuss practice 
issues. 

E3 QA staff report the results of interviews or surveys to quality improvement teams 
or committees. 

§S Quality improvement teams/councils report system strengths and needs to a 
higher level team that takes action to make improvements. 

E3 Regional management, community stakeholders, and state level management 
receive written reports on quality assurance reviews. 

□ Review teams meet with regional management, community stakeholders, and 
state level management to present and discuss the results of their reviews. 

M States require specific, written improvement plans to be developed and QA staff 
monitor progress towards implementing the plans through regular follow up 
reports and/or site meetings. 
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State Example — —Changes in Utah Based on QA Feedback 

Several examples from Utah illustrate the impact of QA feedback on practice and policy 
issues. First, Utah’s qualitative review process has revealed that many staff are comfort- 
able inviting providers to be involved in service planning, but are having difficulty involving 
family members in the child and family services team. During each on-site review, reviewers 
discuss this with caseworkers and supervisors, and find that they sometimes “get the con- 
cept." The regular focus on these issues during the reviews not only emphasizes the agency’s 
commitment to family involvement, but also provides ideas and techniques for caseworkers 
and supervisors on how to make improvements in the future. 

Second, one of the priority focus areas in Utah’s Performance Milestone Plan was prox- 
imity issues, or ensuring that children in out of home care are placed as close as possible to 
their familiar surroundings. Some Regional Ql committees have conducted proximity stud- 
ies, to examine what could be done to improve performance in this area. As a result of its 
study, one region conducted a special recruitment of foster homes for adolescents. Another 
region determined a need for better representation of ethnic minorities among its foster 
homes, and worked with a private foundation to recruit such homes in response. 



Make Improvements 

Some examples of the types of improvements based on quality information, and 
the way they may occur, are the following: 

Improvements in compliance with policy and case practice requirements 
During case review processes, reviewers will point out discrepancies between 
requirements and what is in case records, and work to educate workers on the re- 
quirement and provide assistance to help them improve compliance. For example: 

El During a case record review, if a case record does not include a required perma- 
nency goal, a reviewer would discuss with the caseworker why this had not 
happened and why it is important. In addition, if peers are conducting the reviews, 
the reviewer might share his or her own experience on how these goals are com- 
pleted in his or her own unit. This type of feedback often leads to improvements 
in compliance on the particular case, and in the worker’s other cases in the future. 

H During a qualitative review, if significant parties have not been included in the 
service planning process, the review team would raise this issue with the case- 
worker and supervisor to remind them of the desired practice, discuss obstacles to 
this practice and provide ideas about how to engage parties, or how to develop the 
skills needed to do so. 

Data reports also may point to practice issues that need to be addressed to im- 
prove outcomes. For example, if automated reports from information systems show 
that a particular unit has a much lower rate of family group conferences than other 
units, managers might arrange for enhanced training for caseworkers in the unit, 
and/or expect supervisors to work more closely with caseworkers to improve their 
practice. 




26 



32 



State Example — —Oklahoma's Reports and Improvements 

The Oklahoma DHS quality improvement process includes the capacity to assess service 
outcomes through its SACWIS system as well as through a variety of field based initiatives. 
Oklahoma’s SACWIS system, KIDS, was the first in the nation when it became operable in 
1995, and it produces extensive data. The system produces and distributes over 150 re- 
ports that track basic case data, compliance with policy, the utilization of resources, and 
outcomes. The KIDS office produces data reports, and Program Reid Representatives in 
local areas review data and work with staff to encourage them to make use of this informa- 
tion. In some areas, the field representatives convene meetings of supervisors to review 
their performance on key outcomes and compare their performance to other units and 
offices. Some supervisors use the data to increase awareness among staff of outcomes, 
and to improve practice. For example, areas and units receive a report on total length of 
time to achieve permanency plans, and the report breaks down cases for each worker. 
Supervisors have used this to discuss their unit’s performance relative to other units and to 
highlight individual cases with workers that indicate success or require attention. 



Improvements in documentation 

Review of data reports and case reviews may alert staff to problems in documenting 
their actions through automated systems or case records, or to problems in entering 
data correctly when they do document their action. This often leads to improvements 
in documentation, and subsequently in performance. In the examples above, the 
permanency goal may not have been entered into the record, or the documentation of 
participants in service planning may not have been completed accurately. 

Improvements in policy 

Some issues identified as staff and organizations examine information on quality are 
raised to managers, and have to be addressed by making changes in policy. 

Resource development 

As quality improvement systems examine the performance of child welfare systems, 
it often becomes clear that resources available for children and families are inad- 
equate. The QI process has resulted in initiatives that have increased the numbers 
and types of different kinds of placements, and contracts that have expanded the 
availability of services. 



Evaluate Actions Taken 

After the agency has taken action to make improvements in the child welfare 
system, the quality assurance system needs to evaluate the effectiveness of the 
action. Federal regulations require quality assurance systems to evaluate measures 
taken to address identified problems, and the literature on quality improvement 
confirms the importance of this step in the iterative quality process. For example, in 
the “Plan, Do, Check, Act” cycle, an organization plans an improvement, imple- 
ments it on a small scale, reviews the results and then decides to either implement on 
a larger scale or begin the process over again with more planning (Cassafer, 1996). 
This implies that once a problem has been studied and actions to address the prob- 
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lem have been taken, the process needs to continue. The quality improvement 
process should examine the results of actions taken, and plan further action if neces- 
sary based on the initial impact. For example, if the agency provides additional 
training to staff to improve their practice after an area of weakness has been identi- 
fied, the quality improvement process should reexamine those practice issues some- 
time after the training to see if the action had the intended effect. This continual 
process of checking and revising the actions taken to address identified weaknesses 
is critical to the success of quality assurance systems. 



State Example — —Alabama’s Focus on Improvements 

The multi-faceted QA structure in Alabama and the rich array of information from data, 
case reviews and stakeholder involvement has resulted in positive changes in the system 
and in improvements for the children and families served. For example, County QA Commit- 
tees have made recommendations that have led to improvements in specific cases. Both 
county and state efforts have identified needs for resource development that have been 
addressed. Also, areas where practice needed to be strengthened system wide have been 
identified and steps taken to address them. 

The state also continues to strengthen the follow-up process to ensure that the 
Department’s managers act on recommendations generated by the QA system. For example, 
the state recently added a follow-up process for its on-site county reviews. Four to six weeks 
after an on-site review, the Office of Quality Assurance leads county and state staff and any 
others deemed appropriate ( e.g. community stakeholders, county QA committee members, 
etc.) in a review of the findings. Then the State Conversion Team, comprised of System of 
Care consultants and other state office staff, works with the county to develop or expand 
upon the county’s individualized service plan (ISP). In a process designed to parallel the 
process workers go through with families, the county department must set goals and deter- 
mine the steps they will take within specified timeframes to meet those goals. This ISP will 
be reviewed and updated following subsequent reviews. Additionally, staff from other coun- 
ties provide coaching/mentoring to staff in counties that are moving toward full implementa- 
tion of the QA system. 
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Comprehensive Case Studies of 
State Quality Assurance Systems 



This section presents case studies of three comprehensive quality assurance systems, and pro- 
vides information about components of quality assurance systems in other states. The National 
Child Welfare Resource Center for Organizational Improvement identified the states included in 
this package through recent work and consultation with national experts and state officials. To 
develop these descriptions, we identified the contact in each state who was best able to provide an 
overview of all components of the quality assurance system. Over the fall and winter of 2000-2001, 
we conducted telephone interviews with these contacts, and reviewed written information about 
their quality assurance systems. In some cases we conducted follow up calls to clarify information 
or obtain additional information. We asked each contact to review the written descriptions, and 
revised them based on their comments. 

The three comprehensive systems, in Utah, Illinois and Alabama, have the essential elements of 
quality assurance systems in place. Each agency: 

0 has established outcomes and indicators as part of a planning process and is tracking perfor- 
mance on them 

B has a comprehensive quality improvement processes with staff and others involved in 
examining information on quality in ways that lead to program improvements 

H collects and analyzes information from various data sources, case reviews and stakeholders 
in its quality improvement process, and 

IS has a quality assurance system operating in all of the state jurisdictions where services 
detailed in the child and family service plan are provided. 

These three states provide examples of the multi-faceted quality improvement processes called 
for by the new federal Child and Family Services Reviews. These systems illustrate that states and 
counties can set priorities, examine performance and make improvements in the service systems 
that provide care for children and families in need. 

The case studies highlight three key strategies employed in quality assurance systems: 

H developing and using data on outcomes 

13 using peer review or another effective case review system 

13 implementing effective strategies for stakeholder involvement in assuring quality. 

Specifically, the descriptions highlight Utah’s outcome data, the peer review system in Illinois, 
and Alabama’s strategies for obtaining stakeholder input. After each of these case studies, we list 
other states using that key strategy, including New York, Missouri, Iowa, Oregon, Oklahoma, and 
Texas. These examples provide a broader picture of approaches to using outcome data, conducting 
case reviews, and obtaining stakeholder input. 



By highlighting elements of these states’ quality assurance systems, we aim to facilitate an 
exchange of practice information among agencies. Towards this end, the state descriptions include 
contact information for key staff in each state. We will add examples from other states in the future, 
and welcome input about efforts to improve the quality of services in other states, counties and 
localities. 




30 



36 



Developing and Using Data 

on Outcomes 



Utah 

Utah’s quality improvement system includes clear outcomes and indicators that have been 
established in the state’s strategic plan, and regular tracking and reporting of data on the indicators. 
Utah also gathers information on quality through case reviews and stakeholder input, has developed 
and implemented practice principles, and uses information on quality to make improvements. Key 
features of the system include: 

m Utah tracks and reports performance data on indicators of outcomes established in the 
state’s Performance Milestone Plan. 

m A Department level Office of Services Review (OSR) provides evaluation for the Division 
of Child and Family Services (DCFS). OSR conducts both compliance oriented case re- 
views and outcome oriented qualitative case reviews. 

a Stakeholders participate in qualitative case reviews and serve on statewide and regional 
quality improvement committees. 

® State and regional managers oversee the quality assurance system, and state and local data 
staff produce data and promote its use. A state level trend analysis team also reviews data. 

0 Practice principles have been defined and staff trained on related practice skills. 

a The quality improvement process in Utah has led to changes in practice, improvements in 
documentation, policy changes and the development of new resources. 

Background 

In May, 1999, the Utah Division of Child and Family Services (DCFS) released the Performance 
Milestone Plan, which describes in detail how the Division will improve services to its clients. The 
Performance Milestone Plan evolved out of court involvement in the child welfare system, but the 
Department views it as the business plan, and intends to implement the plan with or without con- 
tinuing court involvement or outside monitoring. Since 1994, the state has been working to comply 
with the monitoring requirements of the “David C. vs. Leavitt” Settlement Agreement. After a 1998 
court order directed the Department to design new, more valid and instructive measures of perfor- 
mance, the Child Welfare Policy and Practice Group (CWPPG) assisted in the development of the 
Performance Milestones Plan. 

The Performance Milestone Plan consists of 9 milestones describing the activities the state will 
engage in to measure progress towards performance goals and make program adjustments based on 
feedback on how well the system is functioning. It defines the Division’s outcomes and the indica- 
tors that will be tracked to measure progress on those outcomes. In addition, the Performance 
Milestone Plan describes the development and implementation of a Practice Model to guide case- 
work practice, defines the structures and processes that will be used to gather and analyze informa- 
tion from data and case reviews, and describes the development of Quality Improvement Commit- 
tees which will use information from these sources to guide necessary changes in the system. Many 
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of the components of the plan have been implemented, and program improvements have resulted in 
the system. 

Data on Outcomes 

One of the milestones in the plan describes the 16 trend indicators that will track the outcomes of 
protection, permanence and well being for those DCFS serves. Data on the trend indicators come 
from the state’s SACWIS system, and an Annual Outcomes Report presents the analysis that re- 
sults. This report uses data on the indicators to measure progress towards specific performance 
goals under nine outcomes in the general categories of protection, permanency and well being. The 
report provides multi-year trend data on each of the indicators, allowing the Division to track 
performance over time. The Annual Outcomes report also includes an appendix that reports on the 
states’ performance on the 13 indicators used in the national Annual Report to Congress, and recent 
trend data on those indicators. Recently, the state also has begun to produce parallel annual reports 
on the regional level, with data on the indicators and outcomes. 

DCFS staff review and use outcome data in two organized ways. The Division has established 
trend analysis teams on the state level who are charged with reviewing and assessing the meaning 
behind the data. The state also has information analysts on the state level who produce the reports, 
and work with data contacts in each region of the state. These regional data contacts have been 
producing reports on the outcomes and indicators in the statewide report on the local level, and 
working to facilitate the use of data. They serve on the Regional Quality Improvement Committees 
to assist in the work of reviewing and analyzing trend data. 

Case Reviews 

Two case review processes measure compliance with requirements and assess the quality of 
services delivered. The Office of Services Review (OSR) manages the reviews. OSR has six staff 
who conduct and manage case process reviews and qualitative case reviews. 

Case process reviews consider the extent to which key case practices are in conformity with 
policy, statute and the milestone plan. These are strictly case file reviews. The Division has stream- 
lined these reviews to focus on 46 case processes, ten of which are judged “critical,” and the other 
36 considered “essential.” The reviews occur annually and examine these case processes in CPS, 
Foster Care and Home-Based cases. Reports go to the regional administrative teams, who discuss, 
among themselves and with supervisors, areas that need to be addressed and strategies for improve- 
ment. 

Qualitative case reviews are intensive reviews of a small sample of cases that involve a review of 
the case record and interviews with all parties involved in the case. In each full year, reviews occur 
on 24 cases in each of the state’s seven regions. Working with a contractor. Human Services and 
Outcomes, Inc., the state has developed a qualitative case review protocol, and staff from both the 
OSR and DCFS have become certified reviewers. In conducting the reviews, OSR involves other 
Department staff, managers and community partners on review teams. The reviews examine spe- 
cific indicators of the status of the child and family, and of the performance of the service system. 
Every qualitative case review generates a narrative that scores these status and system indicators, 
identifies strengths and areas needing improvement, and includes a brief 2-3 paragraph description 
of the issues that need to be addressed. The caseworker, the supervisor and the regional administra- 
tor receive copies of these narratives. 
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The OSR also conducts vexing problem studies, in which they study a problem for which finding 
a solution or answer is difficult. Once the study has been conducted, the Executive Director, DCFS 
management and the DCFS Regional Directors review the results. In addition, OSR staff often will 
review the results of the study with quality improvement committees in the regions. 

Stakeholder Involvement 

Under the milestones plan, the state has established Regional and State Quality Improvement 
Committees. Along with management at the state and county levels, these committees work to 
review information on quality and identify needed improvements. The Regional Committees have 
up to ten members, including two Division staff — the Associate Regional Director and the regional 
data person — and up to eight community members. Community members have a leading role on 
these committees, charged with studying the data and outcomes children, families and communities 
experience, and suggesting changes in resource deployment, policy, procedures and practice that 
will improve or maintain favorable outcomes. Some of the Regional Quality Improvement teams 
have been very active in reviewing information on quality and working for needed improvements. 

Utah obtains input from the children and families served through the qualitative services reviews, 
and involves other stakeholders in conducting these reviews and as members of the quality im- 
provement committees. The state also has created a staff position titled Constituent Services 
Specialist” who is charged with streamlining the process for complaints from consumers and 
working to integrate the recommendations of external review bodies into the ongoing work of the 
Division. This staff member works extensively with regional administrators to assure that issues or 
problems experienced by those served are addressed, and that responses occur at the lowest pos- 
sible level. 

Quality Improvement Staff 

In addition to the state and regional data staff, the trend analysis team, the Office of Services 
Review, and the Constituent Services Specialist, the Division’s managers have a role in quality 
improvement. The milestones plan specifies six priority areas where management will focus their 
attention. Regional Directors have developed regional performance plans that detail how these 
priorities will be addressed. These plans have been shared with QA Committees that work with 
managers to implement the plans. Within the DCFS, the administrative teams of top managers at 
both the state and the regional levels have the responsibility for implementing these plans and 
supporting quality improvement activities. On the state level, the Director for Operations was 
heavily involved in the development of the Practice Model, and a Milestone Plan Coordinator also 
oversees the production of data and implementation of the milestones plan. The state also plans to 
fill positions for Regional Milestone Coordinators (to work with the state level coordinator). 

Practice Standards 

The practice model establishes a baseline defining how the Division interacts with families, 
community partners and staff. The model includes seven practice principles, and a set of key 
practice skills that will be developed through training. For example, one of the principles is part- 
nership,” and related skills are “engaging” children, families and other essential individuals. The 
Division conducted training for all staff on the Practice Model. The qualitative case review protocol 
reflects the practice principles, so the case reviews help communicate to staff the type of practice 
expected under the model. 



Improvements Made 

The process of conducting case process reviews and qualitative services reviews has led to 
changes in practice and improvements in documentation, while the vexing problem studies and the 
work of the quality improvement committees have led to policy changes and resource development. 
Some examples include: 

E3 The review processes often highlight that case records do not accurately reflect what case- 
workers are doing. For example, OSR reviewers have found that some data have not been 
entered accurately because of a coding problem. Utah addressed this issue through training 
workers on the correct way to enter data into the system. 

E The qualitative review process reveals that while many staff are comfortable inviting pro- 
viders to be involved in service planning, they have difficulty involving family members in 
the child and family services team. During the on-site review, reviewers discuss this with 
caseworkers and supervisors, and find that they sometimes “get the concept.” The regular 
focus on these issues during the reviews not only emphasizes the agency’s commitment to 
family involvement, but also provides ideas and techniques for caseworkers and supervisors 
on how to make improvements in the future. 

0 One milestones plan priority area reflected in some regional plans was proximity issues, or 
ensuring that children in out of home care are placed as close as possible to their familiar 
surroundings. Some Regional QI committees have conducted proximity studies to examine 
what could be done to improve performance in this area. As a result of the studies, one 
region conducted a special recruitment of foster homes for adolescents. Another region 
uncovered a need for better representation of ethnic minorities among foster homes, and 
worked with a private foundation to recruit additional homes. 

Utah has many of the key elements of quality improvement systems in place. The system tracks 
specific outcomes, conducts case reviews examining both compliance issues and the quality of 
care, and includes quality improvement committees that involve stakeholders in examining and 
improving the quality of care. In addition, the state has defined practice principles, and trained all 
staff in related practice skills. Finally, many of these sources of information and processes result in 
improvements in the quality of services delivered to children and families. 

Contact: 

Richard Anderson, Director 

Division of Child and Family Services, Utah Department of Human Services 

Salt Lake City, UT 

(801) 538-4100, rianders@hs.state.ut.us 

Linda Wininger, Constituent Services Specialist 

Division of Child and Family Services, Utah Department of Human Services 

Salt Lake City, UT 

(801) 538-4100. lwininge@hs.state.ut.us 
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Other States 

Oklahoma: Oklahoma has a strong culture of quality improvement, and a history of top managers 
committed to the quality improvement process. In the early 1990s, Oklahoma s governor worked 
with private business representatives to establish a quality effort known as “Quality Oklahoma” 
throughout state government. The governor issued an executive order stating that quality is 
Oklahoma’s basic operating principle,” and that “quality means the continuous improvement of 
services to our internal and external customers.” A strategic plan emphasized the need to gather 
data on quality, train all staff on quality, make quality criteria a part of performance evaluations, 
and recognize quality performance. The agency developed a state Quality Council composed of top 
government managers and representatives from the community, and the agency stressed customer 
input and collaborative, problem-solving teamwork (Gunther and Hawkins, 1996). 

This commitment to quality has continued over the years, and the Department of Human Ser- 
vices has been accredited by the Council on Accreditation of Services to Children and Families 
(COA) since the mid 1980s. Today, all staff participate in training on continuous quality improve- 
ment, and managers and staff support assessment, planning and enhancement activities. 

The DHS quality improvement process includes the capacity to assess service outcomes through 
its SACWIS system as well as through a variety of field based initiatives. Oklahoma’s SACWIS 
system (known as KIDS), the first in the nation when it became operable in 1995, produces exten- 
sive data. The system allows the production and distribution of over 150 reports that track basic 
case data, compliance with policy, the utilization of resources, and outcomes. The KIDS office 
produces data reports, and Program Field Representatives in local areas review data and work with 
staff to encourage them to make use of this information. In some areas, the field representatives 
convene meetings of supervisors to review their performance on key outcomes and help them 
compare themselves to other areas. Some supervisors use the data to increase staff awareness of 
outcomes, and to improve practice. For example, the state distributes the report on total length of 
time to achieve the permanency plan to areas and to units, and breaks down results to the worker 
level. Supervisors have used this to discuss their unit’s performance relative to. other units and to 
highlight individual cases with workers (See Attachment #4 Resources Teleconferences). 

The Department also has initiated a variety of assessments of service quality that engage chil- 
dren, families, providers, field and state office staff. The Continuous Quality Improvement Unit, 
which has been significantly expanded in the past year to include a manager and seven professional 
staff, facilitates these efforts. Each county of the state assesses outcomes and systemic factors on an 
annual basis using teams consisting of external stakeholders as well as local and state office staff. 
For a random sample of children, interviews with the child, their family, providers, advocates and 
case worker gather information regarding outcome achievement. Interviews with local community 
stakeholders collect information regarding systemic performance. After reviews occur. County and 
Area Directors should receive completed written reports within a one week interval. Designated 
CQI staff serve as a resource to local offices in efforts to plan for and achieve targeted improve- 
ments in service outcomes. Follow up reviews occur within six months to affirm these initiatives. 

The Department supplements this performance information through the distribution of surveys to 
children, families, providers, staff and stakeholders soliciting feedback regarding their experiences 
with child welfare services. Questions address the realization of key outcomes as well as the respect 
afforded service participants. Staff aggregate findings at the state level on a quarterly basis and 
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immediately forward individual concerns to local offices for action when authorized by the respon- 
dent. Quarterly consultation with an advisory committee consisting of state staff and stakeholders 
helped develop and continues to influence all these efforts. 

Contact: 

Dennis Bean, Programs Administrator 
Continuous Quality Improvement Unit 
Oklahoma Department of Human Services 
(405) 521-4388, Dennis.bean@okdhs.org 

Bill Hindman, Program Administrator 
DCFS Technology & Information Systems 
Oklahoma Department of Human Services 
(405) 767-2527, Bill.Hindman@okdhs.org 



Texas: Texas has a strong data system, and has been using its S ACWIS system — the second oldest 
in the nation — to produce extensive reports for several years. The system produces reports on 
demographics, administrative data and outcomes. Examples include lists of service authorizations 
for clients, the timeliness of initial investigations, and the length of time in placement by perma- 
nency goal. Some of these reports are available at the unit and worker level to help caseworkers, 
supervisors and other managers focus on performance. In some regions, regional managers, pro- 
gram directors and supervisors use the reports extensively to monitor compliance with performance 
standards and to monitor administrative issues such as workload and equity of assignments and 
resources. Each region has an Automation Coordinator who received training from the state and 
provides training to regional staff on accessing and using data from the automated system. 

Contact: 

Grady Rhodes, Project Manager 

Internet Reporting System 

(512) 821-4736, rhodesgw@tdprs.state.tx.us 
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Itrategy #2 



Using Peer Review/Case Review 



Illinois 

A prominent feature of the quality improvement system in Illinois is the peer review process, 
through which staff of the Department of Children and Family Services (DCFS) conduct reviews of 
cases to assess compliance with policies. Peer review is part of the Department’s strategy to involve 
all staff in continuous quality improvement. The Department also tracks and reports outcome data, 
and obtains stakeholder input. The state has dedicated staff working on quality improvement, data 
analysis and overseeing contracted services. Highlights of the system include: 

a All staff in Illinois participate in a continuous quality improvement process and serve on 
quality improvement teams. Local, site and regional quality improvement teams, and a state 
level performance support team, meet regularly to review a whole range of information on 
quality. 

H Through the peer review process, staff not directly involved in the cases conduct regular 
case reviews in every field office. The reviews assess compliance with required policies and 
practices, and aim to promote improvements. 

m Illinois has established outcomes for children and family services and reports data on those 
outcomes. 

a DCFS gathers stakeholder input through surveys at the state and local levels, and through 
child and family services preparatory reviews. 

a The state Division of Quality Assurance includes state level and regional level quality 
specialists and data analysts, and field review staff. In addition, a Purchase of Service 
Division oversees the work of contracted agencies. 

a Illinois has developed organizational support for quality improvement through management 
commitment to the process, the involvement of all staff, and a budget process that includes 
target and actual performance data on key indicators of quality. 

a The quality improvement process has improved practice in individual cases, and improved 
compliance with requirements system wide. 

Background 

Since 1981, Illinois has had a Quality Assurance division which has gathered information and 
produced reports. In the mid 1990s, the Department began to develop a continuous quality im- 
provement (CQI) process stressing the use of information on quality to make improvements. In 
1997, it adopted a quality improvement model with the help of a consultant, Fortena Zerps. Simul- 
taneously, the Department committed to seeking accreditation from CO A, which required develop- 
ing a number of aspects of its quality improvement process (see Attachment #3-National Stan- 
dards) The current CQI process involves collecting data, analyzing it, planning improvements, 
acting to make improvements, and evaluating whether they were effective. 
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The content of state and regional quality improvement plans guides the work. The state’s five- 
year child and family services plan includes the quality improvement plan, describes the compo- 
nents of the QI system that are in place and details what additional elements the state intends to 
develop over the five-year timeframe. Regional managers have developed regional quality im- 
provement plans that help guide the work of managers and staff working on improving quality on 
the regional level. 

Data on Outcomes 

The Dlinois Department of Children and Family Services has established outcomes for the 
Department, and tracks performance on these outcomes regularly. Illinois’ child and family services 
plan describes the Department’s mission, vision and core principles, and the goals and objectives of 
services, defined as measurable outcomes. The Child and Family Outcome Measures Report tracks 
many of these outcomes twice a year on the state level. For each indicator, the report presents trend 
data over multiple years by region. DCFS also produces many other reports on outcomes and 
makes them available at the regional, unit and worker level. For example, the “by worker caseload” 
report lists, for each worker in a unit, the number of children in out of home care by their lengths of 
stay, their permanency goals, and the number who had achieved permanency that month. In addi- 
tion, the state maintains a Performance Outcome and Tracking System (POTS) that allows staff and 
managers to access reports at their desktop. 

State and local level staff examine data. At semi-monthly caseload tracking meetings, central and 
regional office staff discuss and analyze program evaluation data, and focus on improving perfor- 
mance in meeting permanency outcomes. At the local, regional and state level, QI teams examine 
program evaluation data, including outcomes, as part of the standardized QI agenda. 

Case Review 

The field review unit conducts case reviews, and examines compliance with key department 
policies and case practices through an extensive peer review process. Field review staff from the 
Division of Quality Assurance conduct reviews of DCFS operations through on-site audits. The 
staff read and assess a random sample of cases for compliance with department policies and proce- 
dures, accreditation standards, as well as good casework practice. During regional reviews, the 
review team shares findings with operations staff and discusses them in detail. 

Through the peer review process, caseworkers and supervisors conduct reviews of cases in which 
they are not directly involved or have line authority over. In each of the state’s 74 field offices, peer 
review occurs every quarter of each year. The state aims to review 10% of the total cases served 
each year, and occasionally adds extra cases to the sample to ensure that each team reviews at least 
two cases. The peer review instrument focuses on assessing whether key policies and practices 
were followed. The field review unit tabulates results of the peer review and shares them at the 
field office and regional levels. In addition, caseworkers and supervisors for the cases review 
results. If cases do not score well, staff must develop improvement plans for them. 

The state has also begun doing preparatory child and family services reviews that mirror the 
federal review process. These are qualitative reviews that involve reading the case record and 
interviewing all parties involved in the case. The state has conducted these reviews in four regions, 
and intends to have a first round done in all regions by late fall of 2001. DCFS staff, along with 
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staff from other agencies (such as university educational partners) review 120-150 cases in each 
region. The Department contracts with a group at a university to assist with the reviews of con- 
tracted private providers. The reviews focus on assessing outcomes and the service delivery pro- 
cess, and result in written reports. The Division of Quality Assurance then works with the region to 
develop action plans to make needed changes. 

Stakeholder Input 

Illinois gathers stakeholder input through extensive surveys at both the state and local level, and 
also examines data on incidents, accidents and grievances. The state has contracted with the North- 
ern Illinois University to survey foster parents, and the University of Illinois Research Center has 
worked with regional quality councils to survey parents, children and foster parents about their 
level of satisfaction. Local quality teams have access to data on incidents and accidents and reports 
on grievances that have been appealed. In addition, the preparatory child and family services 
reviews provide stakeholder input. 

Quality Improvement Staff 

The state has staff and organizations dedicated to quality assurance and quality improvement 
who draw on the outcome data, case review results and consumer input to evaluate the quality of 
services. The Division of Quality Assurance, headed by an Associate Director for QA and QI, 
oversees the quality improvement process. This Office includes 9 regional quality specialists and 
10 data analysts assigned to the state’s six regions to assist regional staff with the quality improve- 
ment process. A field review unit also has 9 staff, and four program analysts compile data and 
produce reports at the state level. Staff who conduct special projects, and the Office’s managers 
make up the rest of this office. The regional quality specialists and the field review unit both par- 
ticipate in reviews and provide support to other staff involved in the quality improvement process. 
The program analysts who produce data on the state level and the data analysts assigned to regions 
both work to assure that data are available and utilized on the local level. 

Since purchase of service agencies provide over 75% of services, the state has worked to im- 
prove its process of provider review and monitoring. The state created a Purchase of Service moni- 
toring division that consists of the licensing unit, a field audit team, a provider technical assistance 
and training function, and agency performance teams (APTs). The APTs conduct monthly reviews 
of all agencies, reading cases and conducting satisfaction surveys. They also work to develop and 
analyze data profiles of contractor’s services and outcomes. The Purchase of Service Division joins 
the Division of Quality Assurance in providing oversight of the quality of services statewide. 

Organizational Support 

Illinois takes the approach of involving all staff in the quality improvement process. Teams of 
employees take on the role of local quality improvement (QI) teams, and meet at least quarterly to 
review items on a standardized quality improvement agenda. These teams are part of a multi-tiered 
structure of QI teams, where the local teams send representatives to the site QI team, the site QI 
team feeds into a regional quality council, and the regional quality councils are linked to the State 
Performance Support Team. This allows issues to be raised to the appropriate level. 

The items on the quality improvement agenda lead QI teams and councils through a review of all 
of the information on quality — the results of peer reviews, consumer satisfaction data, data on 
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incidents, accidents and grievances, program evaluation data and information generated by special 
projects such as accreditation. The agency director’s commitment to quality and accreditation 
assists the state’s continued work to engage staff in QI. He has made it clear that the quality of 
agency services is a priority within the agency, and he expects all offices and contractors to work 
towards a high standard of service. The Associate Director for Quality Assurance reports directly to 
the Director. 

The state also engages other organizational systems in focusing the agency on outcomes. The 
budget process includes the development of regional management agreements (RMAs), and RMAs 
include performance targets and actual regional performance data on specific indicators. The RMAs 
contain 25 specific measures for safety, permanency and well-being, and also have measures for 
other outcomes (including continuity of care) and system issues. For example: 

0 Under child safety, one indicator in the agreement is the percent of child abuse and neglect 
investigations completed within 60 days. For one region, the target was 98%, the YTD 
actual performance was 96.3%, and the historical baseline for the last two fiscal years was 
86.7% and 90.8%. 

Improvements Made 

Both the peer review and the preparatory child and family review process have resulted in im- 
proved practice in individual cases, and improved compliance with requirements system wide. For 
example: 

h One caseworker described a situation in which a judge wanted to adjudicate a case quickly, 
and asked for the dispositional report within a day. Unexpectedly, the Department was able 
to supply this because the social histories had been completed on time. The staff attributes 
this to the constant vigilance of the peer review process, which is constantly looking at and 
reminding staff about what needs to be done. One of the items peers review and score is 
whether, for intact and substitute care cases, the social history was completed within 30 
days of case opening (See attachment #4 — Resources — Teleconferences). 

■ When the preparatory child and family services reviews uncover safety issues in a case, 
clinical staffings must be held immediately and actions taken to reduce the risk of harm to 
the child. 

The system for assuring quality in Illinois emphasizes continuous quality improvement through 
teams and councils and the involvement of all staff in the QI process. A large state staff produces 
data, helps coordinate different types of case review and stakeholder input, and assists staff in- 
volved in the QI process. 

Contact: 

Michael Sumski, Associate Director for Quality Assurance 

Illinois Department of Children and Family Services (DCFS) 

Chicago, IL 

(3 1 2) 8 1 4-5552, msumski @idcfs.state.il.us 
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Other States 

Missouri: The Division of Family Services in the Missouri Department of Social Services works to 
involve all staff in a Continuous Quality Improvement (CQI) process for child welfare services. 

The CQI process allows the Division to examine agency performance and create plans for ongoing 
improvement. The Division currently is pursuing accreditation with the Council on Accreditation 
for Children and Family Services (CO A), which has promoted the establishment of a comprehen- 
sive approach to the continuous improvement of service quality. This CQI approach involves 
numerous components, including a CQI Meeting Process, Peer Reviews, and Consumer Satisfac- 
tion Surveys. 

On the state level, Missouri has a Quality Improvement Unit comprised of a manager, four 
professional staff, and two support staff who oversee a range of functions related to service quality 
improvement, including the accreditation process and all CQI activities. This unit works closely 
with staff members in the Division’s seven administrative areas who are have direct involvement 
and responsibility for various parts of the CQI process. 

The CQI process intends to empower staff at all levels to be involved in self-directed, self- 
determined change. The CQI Meeting Process involves all staff in the evaluation of the effective- 
ness of the services that the Division provides to children and families. All staff are expected to be 
members of a CQI team that meets regularly to discuss areas needing improvement, develop and 
implement plans to make improvement, and advocate for their proposed improvements. The pro- 
cess includes regular team meetings at a variety of levels, including First, Site, Area, and State 
Level Teams. The multi-level structure provides for the participation of all staff in evaluating 
agency performance and in generating and implementing solutions. In addition to agency staff, 
community members and adult/youth consumers also are involved in the teams at the Site, Area, 
and State Level Teams. All CQI teams must meet at least quarterly, and follow an agenda that may 
include reviews of peer case reviews results; incidents, accidents and grievances; program evalua- 
tion data; strategic plans and training needs; and consumer/personnel surveys. Each of the state’s 
seven administrative areas has a staff person who, in addition to his or her regular responsibilities, 
takes on the role of CQI Coach to assist the teams and the CQI process in their areas. 

The Division has implemented a two-pronged peer review process to assure quality services for 
children and families. These two components include the Peer Record Review (PRR) and the 
Practice Development Review (PDR). Information gained through the two processes provides 
feedback to front-line, supervisory, and administrative staff to assist in the continuous improvement 
of child welfare services. 

The PRR process has been designed to ensure that documentation of essential service compo- 
nents exist in the family record, to provide objective input regarding quality service provision, and 
to identify systemic barriers to quality services. The PRR is an objective process through which 
peer reviewers review randomly selected child welfare case records. Caseworkers and supervisors 
who have had no direct involvement with the case conduct the reviews and focus on assessing 
compliance with key requirements of policy and case practice. 

The peer reviewers utilize a specially designed tool, the Peer Record Review Protocol, for each 
record. A statewide database of data from completed review protocols generates aggregate quarterly 
reports and periodic CQI Teams review them and conduct action planning. Local agency staff 





receive the completed protocols for detailed review once they have been included in the database. 
The agency expects supervisors to share the protocol information with the direct service worker for 
the purpose of developing and implementing action plans for service quality improvement. Also, 
the agency intends the PRR to be constructive in nature and wants findings communicated to case 
managers to promote knowledge and skill development. The state’s goal is to review a minimum of 
10% of the total number of cases served annually. 

For the last three years, Missouri also has conducted Practice Development Reviews (PDR’s) in 
each of the Division’s administrative regions. The PDR is a quality improvement approach devel- 
oped to not only measure the status of children and families served by child welfare, but also to 
assess how effectively other publicly funded service systems (e.g., child welfare, juvenile justice, 
education, mental health, and health, etc.) function to serve children and families. 

The PDR is a professional appraisal of the: 1) status of a child on key indicators; and, 2) ad- 
equacy of performance of essential service functions for that child and his/her caregivers. Each 
child reviewed serves as a unique “test” of child status and as a window into assessing family status 
and related system performance results. A PDR provides a combination of quantitative and qualita- 
tive results that reveal in rich detail what it is like to be a consumer of services and what is working 
at that point in time for a specific child and family. 

PDR activities occur in local communities, and review a representative sample of children 
currently open in child welfare caseloads. The review includes written documentation and face-to- 
face interviews with the focus child (if age appropriate), caregivers, essential family members/ 
significant others, and key service system personnel. Division staff and community partners who 
receive specialized PDR training focused on ensuring inter-rater reliability conduct the reviews. 

The PDR focuses on six core examinations, serving as indicators to determine overall child 
status: safety, stability, physical well-being, emotional well-being, caregiver functioning, and 
learning progress. Concurrently, six core examinations determine overall system performance: 
functional assessment, long-term view, integrated service plan, plan implementation, service coor- 
dination, and tracking/adaptation. The PDR is designed to assess outcomes for children, families, 
and service systems for the purpose of identifying strengths and areas in need of improvement. 

At the end of the review week, the review team produces “core stories” and presents them to 
management and to community stakeholders in separate meetings. The review team also meets with 
the caseworker and supervisor involved in each case to discuss their findings. In addition, the PDR 
aggregates information from individual reviews and reports it in a community feedback meeting at 
the end of the on-site review week. The administration receives a PDR report within two weeks of 
the review. It is expected that a Practice Development Plan be completed within 90 days after 
receipt of this PDR Report. The Practice Development Plan is a strategic plan for the continuous 
quality improvement of local child welfare services. It is a course of action based upon the signifi- 
cant findings of the PDR, building upon identified strengths and addressing identified areas of 
need. 

The PDR process in Missouri is geared towards improving case practice, and the feedback 
provided to caseworkers and supervisors at the end of the review week often leads to improvements 
in practice. For example, some reviews noted that caseworkers were not involving representatives 
from education in the required family support team meetings. As this issue has been raised with 
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caseworkers and supervisors, the state has seen an improvement in how often they are involved, 
and in the positive benefits for children and families. 

Also, see later section on Strategy 3 for a description of Missouri’s strategies for stakeholder 
input. 

Contact: 

Vince Geremia, Manager, Children’s Services Quality Improvement Unit 
Missouri Division of Family Services 
Jefferson City, MO 

(573) 751-8946, vgeremia@services.state.mo.us 



Iowa: Under Iowa’s Quality Service Review (QSR) project, the state conducts qualitative reviews 
of child welfare cases to support learning and practice improvement, create the capacity for self- 
evaluation, and prepare for the federal CFSR process. One staff member on the state level works on 
this project, with the assistance of consultants under a contract with the Child Welfare Policy and 
Practice Group. One additional state staff member produces outcome data and reports and the state 
intends to provide outcome reports and other data that can be used in the quality improvement 
process. Iowa also has an ongoing quality assurance effort that involves case record reviews to 
assess compliance with requirements. 

As in other sites, Iowa’s QSR process involves reading the case record and conducting a series of 
in-depth interviews with all those involved in a case. Typically each team reviews 10 cases in each 
area. The QSR also includes a series of focus groups and stakeholder interviews. A team of trained 
reviewers conducts each case review. The review team consists of a professional reviewer supplied 
by the contractor, and these reviewers often are experienced reviewers from other states. The QSR 
pairs the experienced reviewer with an Iowa staff member who shadows the process, thus gaining 
the experience to qualify as lead reviewers in future QSRs. 

Iowa has allowed local areas to volunteer for a QSR review. As of June, 2001, 8 areas had expe- 
rienced reviews, and 4 more were scheduled for FY ‘02. Volunteer sites also have been allowed to 
choose the method for selecting cases to be reviewed, and many have chosen to focus on specific 
areas — such as CPS cases — or specific groups of cases — such as those that have been particu- 
larly difficult for local staff. On the final day of the review, the QSR team invites caseworkers, 
supervisors, local and regional administrators to attend a feedback session that outlines the review 
findings and opportunities for practice development. 

Within two to four weeks of the review, the supervisors and the area administrator meet with the 
Directors and senior managers, including the Deputy Directors and the Child Welfare Administra- 
tor, to discuss the findings and identify follow-up actions and strategies. The information from each 
review supports local practice development efforts, addresses system performance issues and 
guides the content and instructional design of ongoing training. QSR is designed to be a positive, 
practice development and learning experience. 

The QSR project has focused both on developing good practice at the case level and on learning 
about improvements needed in the organization. For example, the reviews often have shown the 
need to improve engagement skills and increase the use of family team decision making to involve 
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the family in developing and carrying out the service plan. To address these needs, the state will 
expand the training opportunities for staff and supervisors related to engagement skills and family 
team conferencing. The reviews also have highlighted policies that stand in the way of improved 
outcomes for children. The state has addressed these issues where possible by making administra- 
tive rule changes and pursuing cross-department collaborations at the state level. Despite a dramati- 
cally reduced budget due to tax revenue shortfalls, Iowa plans to continue the Quality Service 
Reviews process as part of its ongoing commitment to improve practice, outcomes and services to 
children and their families. 

Contact: 

Jane Kieler, Outcomes Project Director 

Adult, Children and Family Services, Iowa Department of Human Services 

Des Moines, IA 

(515) 281-7233. ikieler@dhs.state.ia.us 



New York City: In New York City, the Administration for Children’s Services (ACS) uses multiple 
approaches to assure the quality of services. ACS has a comprehensive system for ongoing evalua- 
tion of foster care agencies called EQUIP. The Office of Management, Development and Research 
conducts these reviews annually, and they involve an examination of case records, outcomes and 
timeliness of practice. Foster care agencies are rated on their own performance and compared to 
others. The Division of Foster Care and Preventive Services also conducts a case-by-case monitor- 
ing of services. In addition, the Office of Quality Improvement conducts internal quality improve- 
ment through the supervisors conference program, and has recently expanded to work with contract 
agencies through a quality service review process. Under the supervisors conference program, the 
office selects a sample of cases in one office and reviews the case records for documentation. 
Senior supervisors employed by the Office of Quality Improvement then meet with the team of 
supervisors and the manager associated with those cases to review the aggregate data on case 
record documentation for that site, and to discuss quality issues related to practice. Under the 
quality service review process, review teams conduct in-depth reviews of a small sample of cases 
from a contract agency, interviewing all involved in the case, and interview stakeholders. The 
review protocol, developed with a consultant, leads agencies through a self-examination of case 
practice that is an opportunity for learning and for technical assistance on making improvements. 

Contact: 

Diane Connolly, Assistant Commissioner 
Office of Quality Improvement 
New York, New York 

(212) 341-2813, 300R302 @ acs.dfa.state.nv.us 
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Obtaining Stakeholder Input 



Alabama 

Alabama has a strong quality assurance system built on a commitment to involving community 
stakeholders in assessing and improving the quality of child welfare services. A comprehensive 
system is in place to collect and analyze outcome data and to conduct and validate qualitative case 
reviews. Information on quality flows throughout the system and has led to program improvements. 
Highlights of Alabama’s quality assurance system include: 

ii The quality assurance system assesses performance on three outcomes and seven systemic 
factors by gathering and analyzing both quantitative and qualitative information. 

m County quality assurance committees, composed primarily of community representatives, 
meet regularly to review information on quality. The county QA coordinators, with the 
assistance of county staff and input from county QA committees, produce quarterly quality 
assurance reports that compile and analyze information from data and from qualitative case 
reviews. 

m The state supplies county quality assurance committees with data on safety and perma- 
nence, and county staff then supplement it with county level data. 

B County quality assurance committees are responsible for conducting qualitative case re- 
views. 

s The state Office of Quality Assurance, and a State-level quality assurance committee sup- 
port the county quality assurance committees and the county quality assurance coordinators. 
The state Office of Quality Assurance conducts regular on-site reviews of county operations 
that involve qualitative case reviews, an examination of county data for the identified 
reporting period (quantitative information) and interviews with stakeholders. 

to The quality assurance system has resulted in improvements in practice, improved commu- 
nity collaboration/partnership and the development of new resources. 

Background 

As part of the settlement of the R.C. lawsuit in the early 1990s, Alabama agreed to implement 
system reforms throughout the state. The reforms called for the state to implement a quality assur- 
ance system with several specific components. County by county, the state is going through a 
process of “converting” counties to compliance with the requirements of the consent decree and 
working to fully implement the reforms across the state. 

Data on Outcomes 

The whole QA system is geared to examine, report on and improve department performance 
under three outcomes — safety, permanency and well being — and seven systemic factors — commu- 
nity collaboration, service array and resource development, individualized service plans, quality 
assurance and supervision, staffing and caseloads, staff and provider training, and information 
system capacity. The Department has developed 51 indicators for these outcomes and systemic 
factors that the QA system assesses. 
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The system uses several types of information to assess these indicators, including: 

0 Quantitative and factual information to describe activities, service capacity and other mea- 
surable factors 

II Qualitative and outcome information 

□ Information obtained from community stakeholder interviews 

0 Information related to compliance with Federal, State and Department program require- 
ments 

Many staff and organizational units in Alabama help gather and analyze information on quality. 
The system in Alabama has three components: (1) county quality assurance systems, (2) the Office 
of Quality Assurance in the State Department of Human Resources, and (3) the State QA Commit- 
tee. County QA systems consist of a QA coordinator and a county QA Committee. There are 67 
counties in Alabama, and each has an allocated position for a QA coordinator. Counties with popu- 
lations larger than 80,000 have a full time QA coordinator, and smaller counties have a half-time 
position. Coordinators’ responsibilities include organizing and supporting the county QA commit- 
tees, collecting and evaluating information, and issuing the required quarterly and annual quality 
assurance reports. 

Each county has a county QA Committee appointed by the County Director of Human Re- 
sources. These county QA committees primarily include representatives of past service consumers, 
service providers, other public and private agencies, allied professionals, the courts and community 
stakeholders. The County QA coordinator serves on the Committee as a liaison to the agency, and 
occasionally other staff may also join to help integrate QA functions into the Department’s on- 
going service delivery process. 

In Alabama, these community-based QA Committees conduct intensive, qualitative case reviews 
on a minimum number of cases annually and make recommendations back to the Department. The 
county Q A committees also hold regular meetings, routinely review data related to the outcomes 
and systems, and conduct special studies of issues raised by the case reviews and data. The Com- 
mittee also participates by providing input to and approving the county QA reports made to the 
state, and advocates on behalf of the agency, consumers or providers for issues related to improving 
services, agency capacity or outcomes. 

The county QA coordinator with input from the County QA Committee, puts together written 
Quality Assurance reports and makes them available for review by the state Office of Quality 
Assurance, other state office staff, the Federal Court Monitor, the Plaintiff’s attorney and the State 
QA Committee. The Quality Assurance reports provide a means of pulling together data the state 
and counties provide, and for ensuring regular evaluation of data under each of the outcomes and 
systemic factors. The state has an information system that supplies data on safety and permanency. 
Counties supplement this with data on some aspects of permanency and well being, and extensive 
data on the systemic factors that are supplied by the county. For example: 

13 On permanency, the state supplies data on length of stay and permanency goals for children 
in out of home care, while the county supplies the number of children placed in and out of 
county and the number committed to care by different agencies. 

a Counties complete an education data chart on the educational status of children in the 
system. 
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H Counties provide data on staff and provider training, such as the number of staff and provid- 
ers who have completed specific trainings and the number who need training. 

For each of the data items in the report, the county must answer specific questions in a “Discus- 
sion/Analysis of the Data” narrative section. For example, after the data examples we just detailed, 
counties would be required to analyze the data in the following ways: 

0 For children with a length of stay of 13 consecutive months, summarize the barriers to 
achieving permanency and describe efforts underway to ensure that children achieve their 
permanency goals in a timely manner. 

@ Describe the reasons that any children identified as needing special education services did 
not receive such services during the reporting period. 

m Describe any training needs that were identified during the reporting period and how these 
needs are being addressed. 

Case Review 

Both the county quality assurance committees and the state Office of Quality Assurance conduct 
qualitative case reviews. Depending on the size of the counties, county quality assurance commit- 
tees review between 8 and 24 cases each year. They use a written protocol that calls for the review- 
ers to review the case record and interview all parties involved in the case. Prior to conducting the 
reviews, reviewers must be trained in the use of the protocol, and must shadow an experienced 
reviewer and attend the presentation of the case to the Committee. In some counties, QA coordina- 
tors or other Department staff may conduct reviews, but must only supplement the primary role of 
the county QA Committee. 

The state Office of Quality Assurance conducts on-site reviews of County Departments of Hu- 
man Resources. During these reviews, state staff uses the qualitative case review protocol to review 
a small sample of cases and also validates previous county QA Committee reviews by re-reviewing 
a small sample of the same cases. They also interview community stakeholders to assist in evaluat- 
ing the status of outcome and systemic issues, with a particular focus on evidence of a functioning 
County QA Committee. The reviewers use the findings from this review, in conjunction with 
information from the county’s QA report and the outcomes of county QA committee work (e.g. 
case reviews, special studies, etc.), to complete the “Summary of Findings and Recommendations 
Form.” The Summary form reports on the 5 1 key indicators of outcomes and systemic factors, 
highlighting strengths, areas needing improvement and necessary recommendations. 

Stakeholder Input 

Stakeholders participate in assessing the quality of child welfare services both through the 
county quality assurance committees and through a state level quality assurance committee. County 
quality assurance committees receive support from a state quality assurance committee composed 
of representatives across twenty specific organizations and entities that make up the child and 
family service delivery system. The Committee is responsible for monitoring outcomes and agency 
performance from a statewide perspective and for facilitating the development of and networking 
between county QA committees. The State QA committee also serves as a link between the com- 
munity and the State Department of Human Resources, and works to promote an effective child 
welfare system that supports positive outcomes. Specifically, the Committee receives information 



from county QA reports, state QA review reports and statewide QA reports, initiates special studies 
to investigate issues in the reports or information from other sources, and makes recommendations 
for improvements. 

The input of children and families and other stakeholders occurs through the qualitative case 
reviews, and other strategies solicit input from community members. County quality assurance 
committees have used various approaches to soliciting stakeholder input, and state staff interview 
community stakeholders as part of the on-site review. County Departments distribute satisfaction 
surveys, often in conjunction with county QA committees, to gather input from those directly 
involved in cases such as families, children, foster parents, providers and educators. These surveys 
solicit input that is broader than that possible through the qualitative case reviews. The exact 
process for distributing, completing and returning the surveys is not prescribed; however, the 
counties must have such a process in place and the information gained should be used to affirm 
strengths and address areas needing strengthening. 

Quality Assurance Staff/ Structure 

Quality Assurance staff and structures include the county quality assurance coordinators, and the 
county and state level quality assurance committees. In addition, the Office of Quality Assurance 
(OQA) presently includes a Program Manager, Program Supervisor and five Quality Assurance 
Specialists. This state level office supports the State QA Committee, and assists counties in devel- 
oping and maintaining quality assurance functions. They work closely with the one staff person in 
the Office of Data Analysis who produces data from state systems for the counties. The OQA 
analyzes data, and conducts on-site reviews that are used initially to review the extent of the 
county’s conversion to the consent decree principles. Subsequent reviews will determine the level 
of practice and degree of compliance with the Consent Decree (compliance review), or the capacity 
to maintain a high level of practice after conversion (sustainability review). 

Preliminary on-site views occur during the year following the initiation of conversion activities. 
As of May 2001, all counties had begun the conversion process and have had preliminary on-site 
reviews; The timeframe for follow-up reviews depends on the time needed to address the priority 
areas identified in the on-site review. As of August 1, 2001, twenty-six of Alabama’s counties had 
been deemed “converted” by the Federal Court Monitor. For these counties, the Office of Quality 
Assurance will continue to provide training and technical assistance, to review the county’s quality 
assurance reports, and to conduct on-site reviews at five-year intervals. 

Organizational Support 

In settling the R.C. lawsuit, state officials, plaintiffs’ attorneys and experts worked collaborative- 
ly to develop goals for a new system of care, and operating principles that would guide casework 
within the agency. Under the new system of care, individualized service plans that are strengths- 
based and outcome-oriented must be developed with families. The Department is working to train 
all staff on system of care principles. The Alabama Certification Training (ACT training) offered to 
all employees incorporates skills on working with families to develop ISPs. 
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Improvements Made 

The multi-faceted quality assurance structure in Alabama and the rich array of information from 
data, case reviews and stakeholder involvement has resulted in positive changes in the system, and 
in improvements for the children and families the Department serves. 

The County QA Committees have made recommendations leading to improvements in specific 
cases, both county and state efforts have identified needs for resource development that have been 
addressed, and areas where practice needed to be strengthened system wide have been noted and 
steps have been taken. For example: 

a One Chair of a County Quality Assurance Committee described doing case reviews and 
determining that a safety plan existed on paper for a family, but that in practice it was not 
being adequately monitored. Similarly, he described seeing that a case included a perma- 
nency goal, but also needed a plan to include more specific measurable goals that would 
contribute to reaching that long-term goal (See Attachment 4 — Resources — Teleconfer- 
ences). These kinds of observations often lead to improvements in practice as the review 
team and QA coordinator raise the issue with the caseworker and supervisor involved. 

□ State QA reviews often identify and/or support the need for services that are not available, 
or not available in the quantities needed in the county. These issues also surface on the state 
level. For instance, the QA process examines the availability of therapeutic foster homes. 
These resources were deemed as inadequate in earlier reviews, and these findings supported 
some initial movement to provide the resources, including promoting additional state 
contracts for therapeutic foster homes. 

0 The QA process helped state officials recognize the importance of conducting assessments 
with families. One of the performance indicators for the state QA review originally was 
“adequacy of service plan,” and to address the need for thorough assessments, a separate 
indicator, “adequacy of assessment” was added. Since reviewers discuss items in the review 
process with caseworkers and managers, more attention being paid to assessments 

The state continues to work to strengthen the follow-up process so that Department managers act 
on the QA system’s recommendations. For example, the state recently added a follow-up process 
for the on-site reviews. Four to six weeks after an on-site review, the Office of Quality Assurance 
leads county and state staff and any others deemed appropriate ( e.g. community stakeholders, 
county QA committee members, etc.) in a review of the findings. Then the State Conversion Team, 
comprised of System of Care consultants and other state office staff, works with the county to 
develop or expand upon a county individualized service plan (ISP). In a process that is meant to 
parallel the process with families, the county department must set goals, and determine the steps it 
will take within specified timeframes to meet those goals. This ISP will be reviewed and updated 
following subsequent reviews. Additionally, staff from converted counties sometimes will provide 
coaching/mentoring to staff in counties who are moving toward conversion. 

Contact: 

Larry Dean, Program Manager, Office of Quality Assurance 

Family Service Partnership, Alabama Department of Human Resources 

Montgomery, AL 

(334) 242-1472, ldean@dhr.state.al.us 
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Other States 



Missouri: In Missouri, the Division of Family Services (DFS) realized that building a total quality 
organizational culture required feedback from the children and families whom it served. A team of 
participants representing all sections and levels of the organization worked together to develop a 
survey mechanism. As a result, five survey instruments now target: 1) adults recently receiving 
Child Protective Services, 2) adults served through Family-Centered Services, 3) adults served 
through Intensive Family Preservation Services, 4) Foster/Kinship Care providers, and 5) youth in 
Out-of-Home Care. Each survey addresses broad consumer issues such as participation in the 
service delivery process, how consumers feel they were treated, if consumers feel their needs were 
met, and the availability of agency staff. In addition, each survey contains items that address the 
specific needs of each targeted respondent. 

An automated system generates the surveys and DFS sends them via mail at specific points in 
time during and after service delivery. Each mailed survey includes a self-addressed stamped 
envelope to facilitate a higher response rate and ensure confidentiality. Staff enter information from 
all returned surveys into a database, aggregated it, and provide an analysis in report form for review 
in the CQI Meeting process. DFS expects the survey information will assist the agency in respond- 
ing to consumer needs and in providing the best quality services possible. 

Contact: 

Vince Geremia, Manager 

Children’s Services Quality Improvement Unit 

Missouri Division of Family Services 

Jefferson City, MO 65103 

(573) 75 1-8946, vgeremia@services.state.mo.us 
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Attachment 2 

Federal Requirements for Quality Assurance Systems 



Federal law at 471(a)(22) of the Social Security Act requires child welfare agencies to develop 
and implement standards to ensure that children in foster care placements in public or private 
agencies are provided quality services that protect the health and safety of the children. In addition, 
federal regulations require the states to describe in their child and family services plan the quality 
assurance system they will use to assess the services delivered under the child and family services 
plan. Both of these requirements are subject to the child and family services review process. Regu- 
lations describing these requirements are found in two places: 

m Federal regulations on the child and family services plan (CFSP) at 45 CFR section 
1357.15. 

H Federal regulations on the new child and family services reviews (CFSR), at 45 CFR sec- 
tion 1355. 

The quality assurance system is one of the seven systemic factors assessed through the child and 
family services review process, and the two performance indicators assess the state’s standards and 
the quality assurance system (see Table 1) 

Related Quality Assurance Requirements 

There are other requirements in law and regulation that affect elements of quality assurance 
systems. They are related to: 

H Establishing and assessing progress towards outcomes 

S The use of data to measure progress 

0 Consultation with and continued involvement of external partners 

0 Training to support agency objectives 

M CAPTA requirements for citizen review panels and peer review 



These requirements are found in Table 2 (attached) 



TABLE 1 



FEDERAL REQUIREMENTS FOR QUALITY ASSURANCE SYSTEMS 



Requirements in regulation and law 



The state must include in the CFSP a description 
of the quality assurance system it will use to 
regularly assess the quality of services under the 
CFSP and assure that there will be measures to 
address identified problems 
45 CFR 1357. 15(u) 

The State has developed and implemented 
standards to ensure that children in foster care 
placements in public or private agencies are 
provided quality services that protect the health 
and safety of the children 
45 CFR 1355.34(c)(3) 

The state is operating an identifiable quality 
assurance system as described in the CFSP that 

(1) Is in place in the jurisdictions within the state 
where services included in the CFSP are 
provided 

(2) is able to evaluate the adequacy and quality 
of services provided through the CFSP 

(3) is able to identify the strengths and needs of 
the service delivery system it evaluates 

(4) provides reports to agency administrators on 
the quality of services evaluated and the 
needs for improvement and 

(5) evaluates measures implemented to address 
identified problems 

45 CFR 1355.34 (c)(3) 



Performance criteria for systemic factor #3 - quality 
assurance - under child and family services reviews 



The state has developed and implemented 
standards to ensure that children in foster care 
placements are provided quality services that 
protect the safety and health of the children 



The state is operating an identifiable quality 
assurance system that is in place in the jurisdic- 
tions where the services included in the CFSP are 
provided, evaluates the quality of services, 
identifies strengths and needs of the service 
delivery system, provides relevant reports, and 
evaluates program improvement measures 
implemented 

Source: Child and Family Services Review Proce- 
dures Manual, ACF, August 2000 
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TABLE 2 



RELATED QUALITY ASSURANCE REQUIREMENTS 



Establishing and assessing progress towards outcomes 




H The CFSP must specify the goals.. .that will be 
accomplished. ..The goals must be expressed in 
terms of improved outcomes for and the safety, 
permanency and well being of children and 
families, and in terms of a more comprehensive, 
coordinated, and effective child and family 
service delivery system 


45 CFR Section 1357.15 (h), (i), 
and G) 


H The CFSP must include the realistic, specific, 
quantifiable and measurable objectives that will 
be undertaken to achieve each goal. Each 
objective should focus on outcomes for children, 
youth, and/or their families or on elements of 
service delivery (such as quality) that are linked 
to outcomes... 




B The CFSP must describe the methods to be used 
in measuring the results, accomplishments, and 
annual progress towards meeting the goals and 
objectives, especially the outcomes for children, 
youth and families 


45 CFR Section 1355.34(a) 


B [Under child and family services reviews], ACF 
will determine a state’s substantial conformity 
with title IV-B and title IV-E State plan require- 
ments based on 




(1) its ability to meet national standards, set by 
the Secretary, for statewide data indicators 
associated with specific outcomes for 
children and families 




(2) its ability to meet criteria related to out- 
comes for children and families... 


479A of the Social Security Act 


(3) [its ability to meet criteria related to the 
state agency’s capacity to deliver services 
leading to improved outcomes (systemic 
factors)] 




B The Secretary shall. ..develop a set of outcome 
measures.. .that can be used to assess the 
performance of States in operating child protec- 
tion and child welfare programs... and prepare 
and submit to the Congress an [annual] report 
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TABLE 2 



RELATED QUALITY ASSURANCE REQUIREMENTS 



The use of data to measure progress 



S3 ...The state.. .must base the development of the CFSP 

...objectives. ..on an analysis of available baseline information 
and any trends overtime on indicators... 



45 CFR Section 1357. 15(k) 



0 Additional and updated information on service needs and 

organizational capacities must be obtained throughout the five- 
year period to measure progress in accomplishing the goals 
and objectives cited in the CFSP. 



0 ...Processes and procedures assuring the production of valid 
and reliable data and information must be specified [in the 
CFSP], The data and information must be capable of determin- 
ing whether or not the interim benchmarks and multiyear 
timetable for accomplishing CFSP goals and objectives are 
being met 



45 CFR Section 1357. 15G) 



S3 [Under child and family services reviews] states found not to 
be operating in substantial conformity shall develop a program 
improvement plan. The PIP must.. .establish benchmarks that 
will be used to measure the State's progress in implementing 
the program improvement plan and describe the methods that 
will be used to evaluate progress 



45 CFR Section 1355. 



Consultation and continued involvement with external partners 



35(a) 



0 The State’s CFSP must describe the internal and external 
consultation process used to obtain broad and active involve- 
ment of major actors across the entire spectrum of the child 
and family service delivery system in the development of the 
plan. [The consultation process must involve nine specific 
categories described in this section] 

H The CFSP must describe the ongoing consultation process. ..to 
ensure the continued involvement of a wide range of major 
actors in meeting the goals and objectives over the five-year 
operational period of the plan... 

B The full child and family services reviews will. ..be conducted by 
a team of Federal and State reviewers that includes.. .(ii) 
representatives selected by the state, in collaboration with the 
ACF regional office, from those with whom the state was 
required to consult in developing its CFS plan... 



45 CFR Section 1357.15(l)(l) 



45 CFR Section 1357.15(l)(4) 



45 CFR Section 1355.35(a) 
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TABLE 2 - 



RELATED QUALITY ASSURANCE REQUIREMENTS 



Training to support agency objectives 

® The State’s CFSP must include a staff development and training 
plan in support of the goals and objectives in the CFSP... 



45 CFR Section 1357. 15(t) 



US [Systemic factor requirement] The state must operate a staff 
development and training program that (i) supports the goals and 
objectives in the State’s CFSP, (ii) addresses services provided 
under both subparts of title IV-B and the training plan under title 
IV-E of the Act, (iii) provides training for all staff who provide [child 
welfare services] soon after they are employed and that includes 
the basic skills and knowledge required for their positions, (iv) 
provides ongoing training for staff that addresses the skills and 
knowledge base needed to carry out their duties. ..(v) provides 
short term training for current or prospective foster parents, 
adoptive parents, and the staff of state licensed or state-ap- 
proved child care institutions providing care to foster and adop- 
tive children receiving assistance under title IV-E that addresses 
the skills and knowledge base needed to carry out their duties... 



45 



CFR 



Section 



1355.34(c(4)) 



Case Review 



0 [Systemic factor requirement] The state has procedures in place 
that (i) provide, for each child, a written case plan to be devel- 
oped jointly with the child’s parents). ..(ii) provide for periodic 
review of the status of each child no less frequently than once 
every six months by either a court or by administrative review, (iii) 
assure that each child in foster care under the supervision of the 
state has a permanency hearing.. .no later than twelve months 
from the date the child entered foster care...(iv) provide a pro- 
cess for termination of parental rights proceedings. ..(v) provides 
foster parents, preadoptive parents, and relative caregivers of 
children in foster care with notice of and an opportunity to be 
heard in any review or hearing with respect to the child... 

CAPTA Requirements 

ESI Each state receiving a grant for child abuse and neglect preven- 
tion and treatment programs shall establish citizen review panels. 
These panels shall, by examining the policies and procedures of 
State and local agencies and where appropriate, specific cases, 
evaluate the extent to which the agencies are effectively discharg- 
ing their child protection responsibilities... 

0 States receiving community based family resource and support 
grants shall report on the results of the peer review process 
conducted under the state program 



45 CFR Section 1355.34(c(2)) 



106(b)(2)(a)(x), 106 (c ) of the 
CAPTA 
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Attachment 3 

Standards for Quality Assurance Systems Developed 

by National Organizations 



One source of information about the components of an effective quality assurance system is the 
standards related to the management of child welfare agencies developed by national organizations. 
These include: 

m Child Welfare League of America: The Child Welfare League of America’s 1996 Stan- 
dards of Excellence for the Management and Governance of Child Welfare Organizations 
provides a set of best practice standards that are “intended to provide a vision of what is 
best for children and their families”. 

El Council on Accreditation of Services for Children and Families: State child welfare 
agencies are showing an increasing interest in accreditation by national organizations, 
especially the Council on Accreditation of Services for Children and Families (COA). 

COA’s 1997 standards for behavioral health care services and community support and 
education services are intended to be rigorous but realistic descriptions of practice that 
competent private and public organizations can be expected to meet. In 2001, COA is 
releasing revised standards, and the requirements for quality assurance systems have been 
strengthened. 

El National Association of Public Child Welfare Administrators: NAPCWA’s “Guidelines 
for a Model System of Protective Services for Abused and Neglected Children and Their 
Families” describes a model CPS system. 

Information on obtaining copies of these national standards can be found in Attachment #3- 
Resources 

A comparative listing of the quality improvement standards, grouped by the components of child 
welfare systems, is listed is the table on the following pages. Some observations about the national 
standards include: 

B Both CWLA’s management and governance standards and the generic portion of COA’s 
standards require organizations to have a quality improvement process, to use outcomes and 
to do long term and short term planning. 

SI All of the standards call for continuous evaluation, and all of them have standards for 
automated information systems and consumer and stakeholder input 

m COA has specific standards for feedback of information, taking continual action, conducting 
case reviews, involving all personnel, and examining grievances, incidents, or accidents. 
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STANDARDS FOR QUALITY ASSURANCE SYSTEMS 



Component 


CWLA Standards 


2001 Council on 
Accreditation 
Standards 


1997 Council on 
Accreditation 
Standards 


NAPCWA Guidelines 
Protective Services 


Standards 

Planning 


The organization 
should have a strate- 
gic plan that details 
the organization’s 
mission, vision, 
goals, strategies and 
the major actions it 
must undertake in 
the next 3 -5 years 
(2.9) 


The organization en- 
gages in organiza- 
tion-wide, long-term 
planning. ..At least 
every four years the 
organization con- 
ducts a. ..planning 
review that clarifies 
the organization’s 
mission, estab- 
lished goals and 
objectives... and 
identifies and for- 
mulates strategies 
for meeting identi- 
fied goals... (G2.3) 


The process of con- 
tinuous quality im- 
provement includes 
systemic planning... 
the organization de- 
velops a strategic 
planning process for 
long-term visionary 
purposes in which 
longer-term goals 
are established... 
(G2.3) 




Outcomes 


...should translate 
strategic plan into an 
annual operating plan 
that integrates the 
long-range direction 
of the plan into the 
organization’s daily 
activities (2.10) 


Each of the 
organization’s pro- 
grams or services 
annually conducts 
short-term planning 
in support of the 
organization’slong- 
term plan (G2.4) 


The organization... 
establishes short 
term goals which 
support the achieve- 
ment of the organi- 
zational mission 
(G2.3) 






The organization 
should have a written 
statement that sets 
forth its vision, goals, 
and long-term direc- 
tion (2.7) 


The organization 
has an outcomes 
measurement sys- 
tem in each of its 
programs which 
evaluates individual 
progress and pro- 
gram effectiveness 
(G2.7) 


...the organization 
examines outcomes 
of cases. ..[and] ex- 
amines aggregate 
data on outcomes. 
(G2.6) 


Outcome measures 
are needed to evalu- 
ate not only whether 
programs work but 
also how and why 
they work (VI.D) 




ERIC 
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Component 



CWLA Standards 



2001 Council on 
Accreditation 
Standards 



1997 Council on 
Accreditation 
Standards 



NAPCWA Guidelines 
Protective Services 



Quality 

Assurance 

Process 


The organization 
should implement 
and incorporate 
throughout itself a 
process for monitor- 
ing the quality of its 
performance that is 
outcome based and 
centered on the 
child's well-being and 
stability, the... status 
of the child; and the 
achievement of per- 
manence for the child 
(2.77) 


The organization de- 
scribes in writing a 
well-defined process 
for assessing and 
improving its overall 
performance and for 
meeting standards 
that promote quality 
outcomes (G2.1) 


The organization has 
a well-defined and 
designed system to 
assess and improve 
the quality of organi- 
zational perfor- 
mance and to meet 
its standards for 
quality outcomes 
(G2.1) 




Evaluation 


The organization 
should evaluate its 
services at regular in- 
tervals to determine 
whether those ser- 
vices are effectively 
meeting the needs of 
children and their 
families (2.79) 


The organization 
evaluates its sys- 
tems and proce- 
dures and uses its 
findings to improve 
its performance 
(G2.5) 


The organization’s 
systems, procedures 
and outcomes are 
evaluated on an on- 
going basis, the re- 
sults of which are 
used continuously to 
improve perfor- 
mance (G2.6) 


CPS management 
should regularly 
evaluate CPS pro- 
grams, services and 
personnel to ensure 
that all possible re- 
sources are in use 
to provide adequate, 
appropriate services 
to children and their 
families (IV.J) 


Feedback 




The organization 
provides clear, accu- 
rate and timely infor- 
mation regarding all 
aspects of the CQI 
process to its ser- 
vice recipients, 
governing body, per- 
sonnel and other 
stakeholders ...at 
least annually... 
shares findings and 
results; data are 
distributed in a 
timeframe and form 
that are useful to all 
service providers 
and direct service 
staff (feedback 
mechanism) (G2.9) 




All staff must have 
access to timely, 
critical and concise 
information related 
to families served by 
the agency (IV.K) 




62 



64 







Component 



CWLA Standards 



2001 Council on 
Accreditation 
Standards 



1997 Council on 
Accreditation 
Standards 



NAPCWA Guidelines 
Protective Services 



Continual 

Action 




The organization 
takes continual 
action to improve 
services and pro- 
mulgate solutions to 
the issues identified 
by its CQI activities 
(corrective action) 
(G2.ll) 


Continual action is 
taken to eliminate or 
ameliorate problems 
identified in ongoing 
program and out- 
come evaluations 
and in ongoing re- 
views of overall per- 
formance of the 
organization (G2.7) 




Automated 

information 


The organization 
should have a man- 
agement information 
system (MIS) for 
gathering and analyz- 
ing data related to 
strategic planning, 
quality improvement, 
and the evaluation of 
governance, opera- 
tions and program 
services (2.76) 


The organization 
maintains the infor- 
mation that is nec- 
essary to effectively 
plan, manage, and 
evaluate its services 
(G2.10) 

i 


The organization 
maintains informa- 
tion necessary to 
plan, manage, and 
evaluate its pro- 
grams effectively 
(G2.5) 


All staff must have 
access to timely, 
critical and concise 
information related 
to families served by 
the agency (IV. K) 


Case Review 




At least quarterly, 
the organization 
conducts case 
record reviews... a 
sample of open and 
closed cases that 
includes a represen- 
tative sample of 
high-risk cases... 
evaluate presence, 
and clarity... of 
documents. ..and 
appropriateness 
and quality of ser- 
vices (G2.6) 


An internal review of 
randomly selected 
open and recently 
closed cases... is 
conducted on a 
quarterly basis.... 
(G2.6) 

An organization 
which provides... day 
treatment and/or 
residential treat- 
ment services evalu- 
ates the necessity 
of conducting addi- 
tional utilization re- 
view procedures.. 
(G2.2) 
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Component 



CWLA Standards 



2001 Council on 
Accreditation 
Standards 



1997 Council on 
Accreditation 
Standards 



NAPCWA Guidelines 
Protective Services 



Stakeholder 

Involvement 

Staff 

involvement 


Public and nonprofit 
child welfare organi- 
zations should work 
in partnership to en- 
sure the availability 
of a full array of ser- 
vices (1.3) 


Representatives 
from all stakeholder 
groups, including 
persons served, 
personnel from all 
levels of the organi- 
zation, and other 
stakeholders, par- 
ticipate in the CQI 
process (G2.2) 


The participation of 
service consumers, 
all personnel, volun- 
teers and volunteer 
leadership is integral 
to the quality im- 
provement process 
(G2.4) 




Consumer 

input 


Child welfare ser- 
vices should be cus- 
tomer driven, child 
centered, and family 
focused (1.7) 


The organization 
measures con- 
sumer satisfaction 
for all services... 
[through] a con- 
sumer satisfaction 
survey to all per- 
sons served, or a 
statistically valid 
sample... (G2.8) 

At least quarterly, 
the organization 
conducts a review 
of all grievances, in- 
cidents, or acci- 
dents... (G2.5.04) 


Formal and informal 
feedback from con- 
sumers of services 
and other collateral 
sources and stake- 
holders is aggre- 
gated and used to 
improve manage- 
ment strategies and 
service delivery 
processes. . .(G2.4.06- 
2.4.08) 




Organization 
Support for 
Quality 


The organization’s 
corporate and man- 
agement structure 
should facilitate 
movement toward 
fulfillment of the 
organization’s mis- 
sion and achieve- 
ment of the 
organization’s 
vision. (2.68) 

The CEO should pro- 
vide leadership in 
monitoring and evalu- 
ating the quality of 
the services of the 
organization (2.65) 
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Attachment 4 

Resources: Quality Assurance for Child Welfare Agencies 



Child Welfare Outcomes 

“Child and Family Services Reviews: Procedures Manual,” Department of Health and Human 
Services, Administration for Children and Families, Administration on Children, Youth and Fami- 
lies, Children’s Bureau, Washington, D.C., August 2000. Available on the Children’s Bureau 
website at http://www.acf.dhhs.gov/programs/cb or hard copies from the child welfare review 
project at 301-495-1080 ext 3249 or email cw@jbs.l.com . 

The Procedures Manual includes written guidance to states on each part of the child and 
family services review process. Appendices include the instruments for the statewide assess- 
ment and the on-site review and other practical information on organizing the review 
process. Appendix I is the Pathway to Substantial Conformity, which lists the performance 
indicators for each of the outcomes and systemic factors, and provides information on how 
they will be rated. 



“Child and Family Services Reviews: A Resource Book on the New Federal Child and Family 
Services Reviews,” from the National Child Welfare Resource Center for Organizational Improve- 
ment, University of Southern Maine, Portland, ME, February 2001. Available from the National 
Child Welfare Resource Center for Organizational Improvement at 1-800-435-7543 or by email at 
patn@usm.maine.edu 

This package is intended to provide easy access to the documents that provide information 
for states on the new child and family services review process. In addition to selections from 
the Procedures Manual, it also the national standards for the statewide data elements, 
official descriptions of the review process, material produced by states on the reviews, and 
background material on the reviews. 



“Child Welfare Outcomes 1998: Annual Report,” U.S. Department of Health and Human Services; 
Administration for Children and Families; Administration on Children, Youth and Families, August 
2000. Available on the Children’s Bureau website at http://www.acf.dhhs.gov/programs/cb . Copies 
can also be obtained through the National Clearinghouse on Child Abuse and Neglect Information 
at 1-800-394-3366 or by e-mail at nccanch@calib.com . 

The Annual Report presents data on state performance on seven outcomes and thirteen 
associated performance measures. Data in the report is drawn from the Adoption and 
Foster Care Analysis and Reporting System (AFCARS) and the National Child Abuse and 
neglect Data System (NCANDS) 
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“Child Maltreatment 1999: Reports from the States to the National Child Abuse and Neglect Data 
System.” U.S. Department of Health and Human Services; Administration for Children and Fami- 
lies; Administration on Children, Youth and Families, 1998. Available on the Children’s Bureau 
website at http://www.acf.dhhs.gov/programs/cb . Copies can also be obtained through the National 
Clearinghouse on Child Abuse and Neglect Information at 1-800-394-3366 or by e-mail at 
nccanch@calib.com . 

This report presents annual national data about child abuse and neglect known to child 
protective services agencies in the U.S. in Calendar year 1999. The data has been collected 
and analyzed through the National Child Abuse and neglect Data System (NCANDS) 



“Assessing Outcomes in Child Welfare Services: Principles, Concepts, and a Framework of Core 
Indicators,” by the Casey Outcomes and Decision-Making Project, 1998. Available from the project 
website at http://www.casevoutcomes.org 

This publication proposes principles for child welfare services, discusses tools that can be 
used in outcome-based decision-making based on those principles, and presents a set of 
outcome and specific indicators for child welfare agencies to consider. 



WEBSITE: http://ndas.cwla.org 

This is the site of the National Data Analysis System, managed by the Child Welfare League 
of America in cooperation with state child welfare agencies. This comprehensive, interac- 
tive database is integrating national child welfare data from many sources, and makes 
customized tables and graphs available to users through the Internet. 



Quality Assurance - Processes and Tools 

“Evaluation Handbook Series: The Program Manager’s Guide to Evaluation” and companion 
volume, ‘The Children’s Bureau Evaluation Handbook,” U.S. Department of Health and Human 
Services; Administration for Children and Families; Administration on Children, Youth and Fami- 
lies, 1997. Available from the catalog at the National Clearinghouse on Child Abuse and Neglect”, 
1-800- 394-3366 or by e-mail at nccanch@calib.com . 

These guidebooks were developed to assist program managers in assessing program perfor- 
mance and measuring impacts on families and communities. They explain the evaluation 
process and how it may be sued to improve programs. The Children 's Bureau handbook 
addresses evaluation issues specific to the child welfare programs administered by the 
Children ’s Bureau. 
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“Quality Improvement and Evaluation in Child and Family Services: Managing into the Next 
Century,” edited by Peter J. Pecora, William R. Seelig, Fortena A. Zirps, and Sally M. Davis, Child 
Welfare League of America, Inc, Washington, D.C., 1996. Available from the Child Welfare League 
of America, 

This book is designed as a hands on tool to help child welfare agency managers develop 
more data driven, customer focused and outcome oriented approaches to service delivery. 
The tools discussed in the ten chapters include methods of defining services and work 
processes, change strategies, outcome-oriented approaches to practice, program evaluation 
instruments, a quality improvement toolbox, integrated information systems, and break- 
through planning techniques. 



“Using Information Management to Support the Goals of Safety, Permanency and Well Being.” 
Available from the project website at www.muskie.usm.maine.edu/sacwis or from the National 
Child Welfare Resource Center for Organizational Improvement Clearinghouse at (800) Help Kid 
or email clearing@usm.maine.edu 

This training curriculum is designed to teach child welfare supervisors how to incorporate 
information from automated systems into their day-to-day work to support improved out- 
comes for children and families. 



“Total Quality Management in Human Service Organizations,” edited by John Gunther and Frank 
Hawkins, Springer Publishing Company, 1996 

Chapters of this book provide an overview of total quality management as it relates to 
human service organizations, and profile the “Quality Oklahoma ’’ initiative and other 
public sector quality improvement initiatives. 



“Total Quality Management in Human Service Organizations,” by Lawrence L. Martin, SAGE 
publications, Newbury Park, CA, 1993 

This book discusses the features of total quality management and describes the work of the 
primary TQM theorists. It discusses quality improvement tools and strategies, such as 
defining the service delivery system, forming quality teams, techniques for choosing im- 
provement projects, and the need for top management commitment and employee empower- 
ment. 
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Quality Assurance - Teleconferences 

The National Child Welfare Resource Center for Organizational Improvement holds a series of 
audio teleconferences every year to highlight innovative management practices in child welfare. 
Audiotapes and handout packages from the sessions are available from the Clearinghouse at the 
Resource Center ($10 each). Call 1 -800-HELP- KID or e-mail clearing@usm.maine.edu . Recent 
sessions on quality assurance in child welfare are listed below. 



April 11, 2000: “Quality Assurance Systems: Focusing Child Welfare Agencies on Improving 
Quality” 

This session highlights two states that involve a broad range of stakeholders in regular 
reviews of the quality of child welfare services: Illinois, where staff are involved in peer 
reviews, and Alabama, where community-based quality assurance committees conduct case 
reviews. Presenters explore different approaches to quality assurance — both regular re- 
views of a comprehensive range of data from multiple sources, and intensive reviews of a 
small sample of cases as envisioned by the new federal child and family services reviews. 
Presenters: Michael Sumski, Associate Director for Quality Assurance, Illinois Department 
of Children and Family Services and Larry Dean, Program Supervisor, Office of Quality 
Assurance, Alabama Department of Human Resources. 



April 25, 2000: “Outcome Based Management 101 for Supervisors” 

In this session, Resource Center staff joined managers and supervisors to discuss their 
experience using outcome data and its impact on practice at the case level. Presenters: Bill 
Hindman, KIDS Director, Oklahoma Division of Children and Family Services, and Roger 
Ward, Decision Support and Research, Hamilton County, OH 



May 8, 2001: “Building Federal Review Features into Ongoing Quality Assurance Systems” 

This session explores how states can establish quality assurance systems that contain many 
of the same features as the federal reviews — particularly in-depth qualitative case reviews, 
paired with reviews of data and community involvement in assessing agency performance. 
Representatives from Alabama's quality assurance system and from the more recent quality 
service review project in Iowa presented the steps they took to establish ongoing review 
mechanisms and their experience with continually evaluating their own performance. 
Presenters: Larry Dean, Program Supervisor, Office of Quality Assurance, Alabama De- 
partment of Human Resources, and Jane Keilor, Outcomes Project Director, Adult, Children 
and Family Services, Iowa Department of Human Services. 
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May 15, 2001: “Improving Practice, Improving Outcomes” 

This session explores approaches states are taking to improve case practice - both by 
establishing practice principles and standards to guide casework and by reviewing cases 
with the goal of improving practice. State representatives discussed Utah’s practice model 
and Missouri’s practice development reviews. Presenters: Richard Anderson, Director, Utah 
Division of Child and Family Services and Vince Geremia, Practice Development Review 
Manager, Missouri Division of Family Services. 



National Standards for Quality Assurance Systems 

“1997 Standards for Behavioral Health Care Services and Community Support and Education 
Services,” by the Council on Accreditation of Services for Children and Families. Available from 
the Council on Accreditation of Services for Children and Families, Inc., 120 Wall St., 1 1 th Floor, 
New York, New York, 10005, (212) 797-3000 

“COA’s Standards and Self-Study Manual, 7 th Edition, for Public Organizations,” by the Council on 
Accreditation of Services for Children and Families, 2001. Available from the Council on Accredi- 
tation of Services for Children and Families, Inc., 120 Wall St., 1 1 th Floor, New York, New York, 
10005, (212) 797-3000 

“CWLA Standards of Excellence for the Management and Governance of Child Welfare Organiza- 
tions,” from the Child Welfare League of America, Washington, D.C., 1996 . Available individually 
and as a set from the Child Welfare League of America, c/o CSSC, P.O. Box 7816, Raritan, NJ, 
08818-7816, (800) 407-6273 

“Guidelines for a Model System of Protective Services for Abused and Neglected Children and 
Their Families,” National Association of Public Child Welfare Administrators, Washington, 
D.C.,1999. Available from the American Public Human Services Association, (202) 682-0100 or 
pubs@aphsa.org 

These national standards each have specific requirements for quality assurance systems 
( see “National Standards” ). They provide information on what is seen as the model ele- 
ments in strong quality assurance systems 
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Child welfare administrators and senior managers increasingly are searching for new 
ways to assess their agencies’ success in working with children and families. The 
term most often used to describe these efforts is Quality Assurance (QA). In practice, 
“quality assurance” has had no consistent meaning across child welfare agencies. 
Quality assurance efforts may range from administrative case review systems to 
periodic research studies to regular statistical compliance reports to comprehensive 
initiatives involving all these elements and more. 

In this guide, the National Child Welfare Resource Center for Organizational 
Improvement (NCWRCOI) attempts to simplify the picture by presenting a framework 
for child welfare QA systems. While specific QA activities often will vary across 
agencies, the framework includes the broad elements all agencies should consider 
in creating new or energizing existing QA systems. The framework also presents the 
main elements in the form of implementation steps and state examples to assist 
readers in conceptualizing application within their agencies. 
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